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EXECUTIVE SUMMARY 


As part of their improvement efforts, schools have increasingly turned to the use of data to 
improve instruction. This is due in part to the increasing availability of student assessment data 
throughout the school year. The strategy of using assessment and other data to inform teachers’ 
instruction is often called data-driven instruction (DDI). Under DDI, teachers analyze student 
data to better understand students’ learning needs and to identify and improve instructional 
practices to address those needs. 


Research on the effectiveness of DDI is limited. Overall, the research has found that DDI 
does not consistently change what teachers do in the classroom or improve student achievement. 
Different studies of DDI examined different kinds of interventions. The earliest studies examined 
interventions that largely focused on getting data to teachers in a usable form, with less emphasis 
on providing supports for teachers’ data use. This study contributes to a body of more recent 
research examining interventions that placed more emphasis on supporting data use. These 
interventions provided supports to school leaders or teachers on how to analyze data and select 
appropriate instructional strategies through training, coaching, or facilitated collaboration with 
others. 


Specifically, this study examined a DDI intervention that provided substantial training and 
support to school leaders and teachers to help teachers use data effectively to improve their 
instruction and increase student achievement. The study examined the following questions: 


1. How did support for data-driven instruction affect teachers’ use of data and instructional 
strategies? 


2. How did support for data-driven instruction affect students’ achievement? 


Box ES.1. How Was the Study Conducted? 


Study Method: The study used a random assignment design. Schools were selected to participate in the study 
based on their interest in DDI and willingness to be in the study. Schools were randomly assigned to 
the treatment group or the control group. The treatment schools implemented DDI in grades 4 and 5 
from December 2014 through June 2016. DDI effects were measured as differences in outcomes 
between treatment and control schools after 1.5 years. 


Study participants: A total of 102 schools in 12 districts and 8 states participated in the study. The study 
districts were medium to large in size, located throughout the United States, economically 
disadvantaged, and demographically diverse. 


Data Sources: Three types of data were used in the study: (1) information collected from data coaches in the 
treatment group on the frequency and nature of coaching activities through interviews and logs; (2) 
spring 2016 surveys of principals and teachers in treatment and control schools on the supports 
provided to teachers from coaches and school leaders, the frequency and nature of data analysis 
and use, and teachers’ instructional practices; and (3) student-level administrative data on student 
characteristics and achievement in the treatment and control groups. 


Outcomes: The primary outcomes of interest are student achievement in math and English/language arts. 
Intermediate outcomes include: (1) use of data by teachers and school leaders to set and monitor 
student progress; and (2) teacher reports of adjustments to instructional practice to address student 
needs and improve achievement. 
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The Professional Development Program to Support Data-Driven Instruction 
Intervention 


The DDI intervention included two key supports, a half-time data coach for each school and 
consultants from a DDI provider. Consultants provided school leaders and data coaches with 
professional development and ongoing technical assistance. The professional development 
included a two-day introductory session and six subsequent one-day sessions, which mostly 
occurred during spring 2015 so that treatment schools would be ready to fully implement DDI 
during the 2015-16 school year. The ongoing technical assistance began in spring 2015 and 
continued throughout the 2015-16 school year. The intervention also encouraged school-level 
structures and activities to promote and support school leaders and teachers in increasing data 
use throughout the school year. As shown in Figure ES.1, the structures included a school 
leadership team as well as grade-level teacher teams in 4th and 5th grades. 


Figure ES.1. Activities Expected of School Leadership Team and Grade-Level 
Teacher Teams Under Data-Driven Instruction 


School Leadership Team Grade-Level Teacher Teams 


¢ Meet regularly with data coach ¢ Meet regularly with data coach 

e Identify instructional focus for e Jointly analyze student data 
school e Identify and use promising 

¢ Set and monitor school instructional strategies aligned 
achievement goals with guidance from school 

e Provide guidance and support leaders 
on data use to teachers e Iteratively adjust practices 


based on data 


Key Findings 


Most aspects of the intervention were implemented as planned. Hiring data coaches, 
providing them professional development on DDI, and holding regular data-focused meetings 
among school leadership and grade-level teacher teams were all key to implementing the 
intervention as intended. All treatment schools hired experienced educators as half-time data 
coaches, but only 36 percent of data coaches had previous coaching experience and it was rare 
for them to have previously worked as a data coach. Data coaches and school leaders largely 
participated in the expected professional development; depending on the session, participation 
ranged from 87 to 97 percent for data coaches and 75 to 89 percent for principals. Even so, about 
one-fifth (22 percent) reported that their training either did not prepare them or prepared them to 
carry out some but not all data coach tasks by the end of the study period. 


Principals were expected to meet weekly with data coaches to strategize for DDI 
implementation, allocate time and resources to support DDI, and set expectations and provide 
guidance on data use by teachers. Most principals (59 percent) met with data coaches at least 
three times per month to monitor and support DDI, and most (at least 66 percent, depending on 
the type of guidance) provided overall guidance on DDI. Within this context, more teachers in 
treatment than control schools worked one-on-one with a data coach or school leader on data- 
related activities; for example, 69 percent of treatment teachers compared with 56 percent of 
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control teachers reported at least monthly one-on-one interactions on analyzing or interpreting 
student data. Most treatment schools also established grade-level teams of teachers that met 
regularly with the data coach (77 percent of treatment teachers reported at least monthly 
guidance on using and analyzing data compared with 61 percent of control teachers). More 
teachers in treatment than control schools reported receiving coaching on data-related activities 
in collaboration with other teachers during common planning periods; for example, 52 percent of 
treatment teachers compared to 34 percent of control teachers reported at least monthly training 
or coaching on how to analyze and interpret student data. 


Despite additional resources and emphasis placed on data-related activities, the 
support for data-driven instruction intervention did not increase key data-related 
activities. Teachers were expected to work together in teams to examine data in order to 
understand individual student needs at least twice per month. However, similar percentages of 
teachers in treatment and control schools reported data-related activities during common 
planning time; for example, 79 percent of treatment teachers and 73 percent of control teachers 
reported jointly analyzing data to understand student needs at least monthly. These activities 
were fairly common among teachers in control schools even without the additional resources 
provided as part of the DDI intervention. 


The support for data-driven instruction intervention did not increase teachers’ data 
use or change their instructional practices. The intervention was intended to increase data 
analysis and collaboration with other teachers, which would then lead teachers to change their 
instructional practices in order to appropriately address student needs. Because the intervention 
did not increase data-related activities, it is consistent that similar percentages of teachers in 
treatment and control schools reported using each of nine data practices. For example, 38 percent 
of treatment teachers and 35 percent of control teachers reported monitoring student progress, 
and 43 percent of treatment teachers and 44 percent of control teachers reported planning 
individualized instruction, daily or several times per week, in math. There were also no 
treatment-control differences in English/language arts. Nor did the intervention lead teachers to 
report more frequent use of any of five instructional practices potentially associated with DDI 
that were examined (figure ES.2). 
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Figure ES.2. Percentage of 4th and 5th grade teachers who used instructional 
practices daily or several times per week during 2015-2016 


Percentage of Teachers Who Report Using Strategy 
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Source: Teacher survey (n = 397-411). 
ELA = English/language arts. 
*Difference is statistically significant at the .05 level, two-tailed test. 


The support for data-driven instruction intervention did not affect students’ 
achievement. On average, 4th and 5th grade students in treatment and control schools had 
similar achievement in math and English/language arts (figure ES.3). Students in each group 
scored at about the 40th percentile on state assessments in each subject, on average. The study 
also found that support for DDI did not improve achievement for any subsets of students 
examined, such as students with different prior achievement levels. Nor did the study find that 
DDI consistently improved achievement for any of the subsets of schools examined, such as 
schools with greater readiness to implement DDI. 


Figure ES.3. Mean student achievement on 2016 state assessments in math 
and English/language arts 
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Source: District student records (n = 12,018-12,036). 
Neither difference is statistically significant at the .05 level, two-tailed test. 
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Concluding Thoughts 


This study is part of a growing body of research on how DDI affects teacher practices and 
student achievement. While early studies focused on the effects of giving teachers greater access 
to data, this study contributes to more recent evidence focused on the effects of giving teachers 
more support in their efforts to use data to improve instruction. The DDI intervention being 
examined in this study was designed to give teachers support from a half-time data coach, 
actively engaged school leaders, and fellow teachers in a group setting. 


The treatment schools implemented most aspects of DDI as planned, and teachers in these 
schools reported receiving more support from coaches and school leaders than those in control 
schools. But support for DDI did not increase teachers’ data use or change how often they used a 
set of instructional practices related to DDI. Most importantly, support for DDI did not affect 
student achievement. These findings are similar to those of other recent studies. As a strategy to 
improve student achievement, DDI relies on: (1) using a data-focused approach to identify areas 
for instructional focus, (2) finding appropriate strategies to improve instruction, and (3) 
implementing teacher practices that improve student performance in the focused area. This 
study’s findings in conjunction with other recent studies suggest that simply giving teachers 
more support in their data use is not sufficient, on its own, to improve student achievement. 
Future research might turn to other possible ways of improving DDI interventions, such as 
improving the quality of data-use support provided to teachers or changing the nature of this 
support by placing greater emphasis on how teachers use data to identify and implement 
effective instructional practices. 
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I. INTRODUCTION 


As part of school improvement efforts, educators have increasingly turned to the use of data 
to improve their instruction. Although federal policy does not require schools to use data 
regularly to make instructional decisions, it does indirectly encourage the use of student 
assessment data in school accountability efforts. For example, the Every Student Succeeds Act of 
2015 requires each state to annually assess and publicly report on multiple measures of school 
success, including academic indicators for all students and subgroups of students, based in part 
on student assessments. The U.S. Department of Education has also provided guidance on data 
use in schools with a What Works Clearinghouse practice guide, Using Data to Support 
Instructional Decision Making (Hamilton et al. 2009). 


The increasing use of student assessments throughout the school year has also contributed to 
a rise in the use of student data in school improvement efforts (Datnow and Hubbard 2015; 
Datnow and Park 2014; U.S. Department of Education 2010). These include less formal 
assessments—homework assignments, quizzes, or end-of-unit tests—and more formal interim 
assessments—standardized tests administered at different points throughout the year and 
designed to measure mastery of key standards covered in the curriculum. 


The literature suggests the value of data use, but the evidence is not strong. Among high- 
performing charter schools, for example, frequent data use is a common feature (Hoxby et al. 
2009; Angrist et al. 2013; Dobbie and Fryer 2013; Chabrier et al. 2016). However, little rigorous 
research exists about whether formal efforts to promote teachers’ data use lead to better 
instruction and improved student achievement. 


Data-driven instruction (DDI) involves the use of data by schools and teachers to inform 
teachers’ instructional practices. Because the student data can show different levels of mastery 
on different topics and among different students, teachers can tailor their instruction accordingly. 
For example, Hamilton et al. (2009) describe changes to instruction that teachers might make 
under DDI, including (but not limited to) spending more time on certain topics, differentiating 
instruction by grouping or regrouping students to receive additional help in certain areas, or 
using new ways of teaching difficult material based on best practices identified by teaching 
colleagues. DDI interventions are systematic efforts to ensure that teachers have access to the 
right kinds of data and support in using the data effectively to improve their instruction. 


This study examines one such DDI intervention, designed to train and support teachers and 
school leaders in using data to improve instructional practices and, ultimately, student 
achievement. The study has two goals. The first is to examine whether the intervention was 
implemented in schools as expected based on the intervention design. The second is to test 
whether the intervention led to changes in teachers’ instructional practices and improvements in 
student achievement. 


Previous Research on DDI Interventions 


Although the literature on DDI is growing, no consensus exists on its effectiveness in raising 
student achievement or which features of DDI interventions are most important. The most 
rigorous studies have used an experimental design, in which implementation of a DDI-focused 
intervention is randomly assigned to one set of schools or districts, which are then compared 
with another set randomly assigned to not implement DDI. 
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The experimental research on DDI illustrates that the nature of the interventions being 
studied has evolved over the period covered by the studies. The early studies (Carlson, Borman, 
and Robinson 2011; Cordray et al. 2012; Slavin et al. 2013; Konstantopoulos, Miller, and van 
der Ploeg 2013; Konstantopoulos et al. 2016) examined DDI interventions implemented before 
2010, which focused primarily on providing schools with interim assessments, along with 
software that produced reports summarizing assessment data in ways that teachers could easily 
understand. By contrast, later studies were conducted at a time when it was more common to 
administer interim assessments and for teachers to have access to the data. These studies 
(Cavaluzzo et al. 2014; West, Morton, and Herlihy 2016) examined DDI interventions that 
focused on supporting teachers in their efforts to interpret and use the data to improve their 
instruction. Examples of these supports included the following: 


e Periodic data coaching for teachers by an external provider or instructional specialist to help 
them analyze student data and understand what the data mean about their students’ 
understanding of key concepts and where the teachers’ instructional practice may and may 
not be succeeding 


e Engagement of school leaders to ensure that teachers have sufficient time and resources to 
work with the data and provide them with guidance and feedback on what sources of data to 
use, how to analyze the data, what aspects of the curriculum on which to focus, and what 
evidence-based instructional strategies to consider using 


e Professional learning communities or other teacher collaboration around data use so 
teachers can support one another in understanding what the data mean regarding their 
students and what sorts of instructional strategies might be successful in these situations 


The experimental research provides little evidence that DDI leads to changes in teachers’ 
instructional practices. None of the three studies that examined teachers’ use of differentiated 
instruction or other instructional practices found significant positive impacts of DDI 
(Konstantopoulos, Rafiullah Miller, and van der Ploeg 2013; Cordray et al. 2012; West, Morton, 
and Herlihy 2016). Two studies did find that DDI led to more data use among teachers 
(Cavaluzzo et al. 2014; West, Morton, and Herlihy 2016). However, it was not clear what 
changes—if any—occurred in their classroom instruction as a result. 


This research also provides no consistent evidence that DDI leads to improved student 
achievement. Cordray et al. (2012), Cavaluzzo et al. (2014), and West, Morton, and Herlihy 
(2016) each found that the DDI intervention did not have a statistically significant impact on 
student achievement. Other studies reported mixed findings. Konstantopoulos, Miller, and van 
der Ploeg (2013) and Konstantopoulos et al. (2016) found negative impacts of DDI among 
younger students (K-2) for one cohort and nonsignificant impacts for the other cohort; among 
older students (grades 3-8), they found positive impacts for one cohort and nonsignificant 
impacts for the other. Carlson, Borman, and Robinson (2011) and Slavin et al. (2013) each 
examined a single DDI intervention and found positive impacts in some grades and years, but not 
others. 


Several of these studies note that the lack of consistent student impacts may be because 
teachers were unable to take the step from analyzing data to changing instructional practices or 
differentiating instruction for struggling students (Cordray et al. 2012; Slavin et al. 2013; 
Cavaluzzo et al. 2014). Several also indicated that these DDI initiatives were not the only source 
of support for teachers, so it might be that the availability of professional development or other 
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ongoing assistance may have reduced the possible influence of the DDI program the researchers 
examined (Cordray et al. 2012; Konstantopoulos, Miller, and van der Ploeg 2013; 
Konstantopoulos et al. 2016). Other studies indicated that DDI programs may act more in the 
long-term, so the one- to two-year studies did not observe impacts (Cavaluzzo et al. 2014; 
Konstantopoulos, Miller, and van der Ploeg 2013). Relatedly, West, Morton, and Herlihy (2016) 
found that impacts were positive for schools that had the right structures in place and where 
leadership was ready to prioritize the work, but there were no or negative impacts in other 
schools. This suggests that a DDI program might not be successful in a broad set of schools but 
could be successful in schools that were most ready from the outset to put the program in place. 


Taken as a whole, previous experimental research on DDI interventions is limited, but 
suggests that past efforts to provide teachers with assessment data and in some cases support 
their data use have not been sufficient to change what the teachers do in the classroom and, 
ultimately, improve student achievement. However, most of the DDI interventions studied thus 
far have focused primarily on getting data to teachers in an accessible form, and less on 
supporting their use of data to improve instruction (Carlson, Borman, and Robinson 2011; Slavin 
et al. 2013; Konstantopoulos, Miller, and van der Ploeg 2013; Konstantopoulos et al. 2016). The 
early interventions studied previously provided little direct support to teachers beyond 
professional development around the analysis of data from the specific assessment being 
evaluated. 


DDI interventions examined in more recent studies incorporated some supports to teachers 
but perhaps not as intensively as needed. Cavaluzzo et al. (2014) studied the Using Data 
initiative, which included school-based instructional specialists as data coaches and emphasized 
teacher collaboration, but did not systematically engage school leaders to support teachers’ data 
use. The Achievement Network program studied by West, Morton, and Herlihy (2016) included 
using data coaches and engaging school leaders to support teachers’ data use. However, the 
Achievement Network data coaches’ main emphasis was to “build school leaders’ capacity to 
support instructional data use in their schools” rather than directly coach and support teachers in 
their data use. The Achievement Network program also did not explicitly include teacher 
collaboration within schools.! 


In this study, we hypothesize that a key to the success of DDI may be not only to provide 
formal professional development on data use but also to ensure that teachers get intensive data- 
related support from multiple sources—data coaches, school leaders, and other teachers who are 
peers. In other words, we assume that teachers need additional support in using data to 
understand the needs of their students and improve their instruction, and are not already using 
data effectively (Means et al. 2011). The hypothesis motivating the study also assumes that most 
teachers already have access to student data, including assessment data, attendance, and 
examples of student work. Thus, they do not need additional forms of student data as part of a 
DDI intervention. Instead, what they really need is help and support in figuring out what to do 
with the data they already receive. The DDI intervention in this study was designed to provide 
such support in the study’s treatment schools. As a result, the study is designed to test whether 
providing intensive data-related support for teachers is the key for DDI to improve teachers’ 


' There have also been studies of DDI using nonexperimental designs, which generally provide less rigorous 
evidence than experimental studies. Consistent with the experimental evidence, several non-experimental studies 
found little impact of DDI on student achievement (such as Quint et al. 2008; Henderson et al. 2007). 
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instructional practices and students’ achievement. This study does not examine the effects of 
providing teachers with additional forms of student data. 


Overview of the Support for DDI Intervention Examined in This Study 


Taking into account the nature of previous DDI interventions, findings from the literature 
review, and input from a technical working group, this study examines a distinct DDI 
intervention. The key aim was to create the conditions within participating schools that would be 
most likely to foster success: coaching for teachers on data use, active support from school 
leaders, and collaboration among teachers. The study team subcontracted with Focused Schools 
(formerly Focus on Results), an organization that works with districts and schools in the United 
States and Canada on DDI and school reform, to implement this DDI intervention in study 
schools.” The intervention was implemented for roughly a year and a half, from December 2014 
through June 2016. 


A logic model (figure I.1) illustrates the intervention’s supports, services and activities, and 
intended effects. The DDI intervention involved two key supports: a half-time data coach for 
each school and consultants from Focused Schools who worked with the schools. Together, these 
individuals engaged in two types of activities and services intended to build schools’ capacity to 
conduct DDI: (1) professional development and technical assistance for data coaches and school 
leaders, and (2) targeted school-level activities designed to support school leaders and teachers in 
increasing data use. The assistance to school staff was based on an underlying assumption that 
they needed more than just access to data and analysis tools. The staff also needed help and 
support in the process of analyzing data to better understand their students’ needs, and in 
identifying and implementing the instructional practices that might best address those needs. The 
study covered the costs to schools of the DDI training and technical assistance, as well as the 
salaries of the half-time data coach for each treatment school for two full years. This report 
describes the DDI intervention’s supports, services, and activities in detail below. 


The study team’s hypothesis was that these supports, services, and activities, if implemented 
with fidelity, would improve teacher practices, in turn improving student achievement. In the 
near term, fourth- and fifth-grade teachers were expected to make greater use of student data to 
set and monitor improvement goals, and identify promising instructional strategies. If the 
intervention engaged school leaders appropriately, school principals’ data use was also expected 
to increase. 


The DDI intervention did not require that teachers implement specific instructional 
approaches, but they were generally expected to make greater use of evidence-based 
instructional strategies, such as reviewing and adjusting students’ small-group assignments, 
using differentiated instruction, re-teaching difficult material, and increasing time spent on 
instruction. However, the idea was that the specific instructional strategies teachers used would 
be determined by the study data. If the data indicated that most students were struggling on a 


? The intervention was based on Focused Schools’ usual approach toward assisting schools, in terms of the content 
and activities of the professional development and technical assistance. These activities were put into a package that 
could be implemented during the evaluation period. Focused Schools developed professional development materials 
(or used existing materials) that set expectations about what schools needed to do to put the intervention into place, 
with the objective that schools implement key aspects of the intervention by fall 2015 so that treatment schools 
would have the full intervention in place for nearly the entire 2015-2016 school year. 
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particular topic, for example, teachers might make changes to instruction that would affect all of 
their students (such as re-teaching a particular lesson or skill using a different strategy). 
Alternatively, if the data showed that subsets of students were having difficulty in a particular 
area, teachers might group students based on their mastery of the concepts in that area and use 
different strategies for these different groups—that is, use differentiated instruction. They might 
provide additional instruction on these subjects for the struggling students. If the data helped 
teachers use more effective strategies for the students in the class who most needed help, the 
team hypothesized that the students would subsequently perform better in math and 
English/language arts. 
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Figure I.1. Logic model for the support for DDI intervention 


Participation and Amount 


NCEE 2019-4008 


Professional Development 


* Orientation session to introduce DDI and describe 
coach and principal roles and activities 


Professional 
Development (PD) and 
Technical Assistance 

(TA) Providers 
(“consultants”) from 
Focused Schools 


Ongoing PD sessions to train instructional leadership 
teams on DDI activities, including guidance & 
materials to help them train teachers 


Technical Assistance 


¢ Consultant provide customized TA to coaches, 
Data Coach principals, and other school leaders 
(one half-time coach 


per school) Assess how implementation is going and give 


feedback through visits, calls, email, and online 
discussion board 


Activities in Schools 


Leadership from Principals 
+ Set data use expectations for teachers and school leaders and monitor/provide feedback 
- Support, strategize, and set priorities with data coach 


Formationof School Working Groups 

Instructional Leadership Team (ILT) 
Analyze schoolw ide data and formulate student achievement goals 
Identify w eaknesses w ith teachers’ core instruction to determine an instructional focus, as 
wellas promising evidence-based instructional strategies to address these w eaknesses 
Plan PD and monitor progress tow ard student achievement goals 


Teacher Collaboration Teams (TCTs) 

* Work collaboratively to implement DDI through structured activities 

- Analyze and interpret student data to set learning goals 

+ Identify and plan forimplementation of evidence-based instructional strategies to help 
students meet learning goals, and monitor progress 


Assistance from Data Coach 

+ Lead and support DDI implementation on day-to-day basis (e.g., lead/co-lead ILT and TCTs) 
- Analyze student data and create data walls 

+ Support teachers in their DDI workby meeting one-on-one and observing their classes 


Professional Development 

- Single 2-day orientation session 
for coaches and principals, 
December 2014 
Six 1-day sessions per district for 
schools’ ILTs, January through 
October 2015 

Technical Assistance 
Up to 8 in-person visits/district 
Monthly calls & other contacts as 
needed 


Online log/discussion board for 
coaches and consultants 


Recommended 
Participation and Amount 


Leadership from Principals 
- Weekly meetings with coach 
* Ongoing support 


School Working Groups 

ILT 

* Data coach, principal, 4th & 5th 
grade chairs, other leaders 

+ Meet once or twice a month 


TCTs 

* Separate 4'r & 5 grade teams 

- Data coachand all teachers in 
grade 

- Meet twicea month 


Assistance from Data Coach 
* Ongoing support on as-needed 
basis covering half time 


Final 
Outcomes 
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Student Achievement in 
Mathematics and English/ 
Language Arts 


Intermediate 
Outcomes 


Increased Use of Data by 

Teachers and School 

Leaders 

* Teachers make greater 
use of student data to 
set and monitor 
improvement goals and 
identify promising 
instructional strategies 


Im proved Teacher 

Instructional Strategies 

+ Teachers make greater 
use in their classroom of 
evidence-based 
strategies to address 
student learning needs 
and improve instruction 
(e.g., differentiated 
instruction, adjust 
student groups, increase 
time in low -performing 
areas) 
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Focused Schools Consultants’ Services 


Consultants from Focused Schools provided services through professional development and 
technical assistance. Both types of services reflect the intervention’s train-the-trainers approach, 
in which the trained data coach and school leaders would then work with teachers to use DDI 
techniques to inform their instructional practices. 


The overall focus of the professional development and technical assistance was to help 
schools put in place and effectively support a central feature of DDI—the collaborative cycle for 
data analysis. The initial stage of the cycle involves setting schoolwide or grade-level student 
achievement goals and analyzing data to assess progress toward those goals. Careful 
interpretation of this data analysis allows teachers and school leaders to diagnose the specific 
topics of concern or standards that students are struggling to grasp, as well as whether they are 
problem areas for all or particular groups of students. This approach allows them to identify 
current instructional practices that are not working as well as they should and need to be 
improved, and to collaboratively identify evidence-based instructional practices to address these 
problem areas. With the support of the coach, teachers then implement these practices and 
monitor the results of the implementation by again analyzing student data, and the cycle then 
repeats. 


Professional development. Professional development began with a two-day orientation 
session in December 2014. The session introduced data coaches, principals, and selected district 
administrators to DDI, including the approach to using student data to improve instruction, the 
roles they would play in the intervention, and prepared them to begin implementing DDI in their 
schools. 


The intervention also included ongoing professional development sessions. The sessions 
were intended to deepen school leaders’ understanding of DDI and help them implement it in 
their schools. Specifically, the ongoing professional development aimed to guide data coaches 
and other school leaders in their analysis of student data (Box 1) and, based on this analysis, their 
approach to selecting evidence-based instructional strategies (Box 2). Six one-day sessions were 
held in each district—four in spring 2015 and two in fall 2015. Each session focused on a 
different DDI topic (table I.1). All instructional leadership team members from each school were 
expected to participate in all of the sessions. The attendees typically included the school’s 
principal, data coach, chair or lead teacher in fourth and fifth grades, and other school leaders. 
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Box 1. What were schools trained to do with student data? 


Focused Schools’ approach to training professional development participants was intentionally simple and 
straightforward, based on a belief that this approach would be most likely to lead to successful implementation. 
The basic approach involved dividing students into three color-coded groups based on their understanding of a 
particular concept or standard: those who were proficient (green), those who were approaching proficiency 
(yellow), and those who were well below proficiency (red). This analysis approach was intended to be applied to 
different types of data to shed light on student understanding of different topics or standards—for example, to a 
single component of a formal interim assessment. It also could be applied to more informal formative 
assessments, such as quizzes or homework assignments. Dividing students into these groups for different topics 
or standards ideally would help schools identify specific areas of weakness, either across the school (or grade 
level) or for individual teachers. 


The Focused Schools’ professional development materials guided teachers to look at achievement data at the 
group level—the grade level or school as a whole—as well as at the individual student level. Focusing on groups 
would help schools identify issues with their core instructional practices. Proficiency rates that were high for most 
standards but consistently low for one or two of them (and these standards were covered in the curriculum) might 
suggest a weakness in instruction in that area. At the individual level, the green/yellow/red data analysis would 
help individual teachers target any changes in their instructional practices to specific concepts and to those 
students most in need of help. The materials also encouraged teachers to analyze data separately for key student 
subgroups (such as those in different socioeconomic categories) to identify any achievement gaps. 


Finally, the professional development materials encouraged schools to look at student work, and not only at data 
indicating whether students mastered particular concepts or answered questions correctly. By examining student 
work collaboratively, teachers may better understand how their students approached their work or what obstacles 
may have prevented them from mastering the content. A major section of one session was devoted to this topic, 
guiding participants through an example of “looking at student work.” The basic approach here mirrored the 
approach discussed above, focusing on understanding differences between the work of students at different 
levels of proficiency. 
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Box 2. What information was given to schools on evidence-based strategies 
to improve instruction? 


Focused Schools encouraged schools to use evidence-based instructional strategies in two ways. First, a major 
focus of one of the professional development sessions involved selecting evidence-based instructional strategies. 
Materials from that session included a summary of instructional strategies with some research evidence of 
effectiveness. One source of possible instructional strategies teachers might operationalize was drawn from the 
work of John Hattie, Professor of Education and Director of the Melbourne Education Research Institute at the 
University of Melbourne, Australia (Hattie 2009). ' 


Second, Focused Schools encouraged data coaches to compile additional resources related to evidence-based 
instructional strategies on their own. As part of the implementation of the intervention in schools, data coaches 
were to develop resource rooms for fourth- and fifth-grade teachers. These rooms might include materials that 
help teachers understand data, identify evidence-based practices (for example, Hattie’s research), or include 
professional development materials on implementing specific instructional strategies. 


Overall, the intervention placed particular emphasis on two instructional strategies—differentiated/small group 
instruction and maximizing instructional time. For example, one strategy mentioned in the training materials to 
maximize instructional time was “bell-to-bell teaching,” to reduce any dead time during blocks devoted to key 
areas of instruction. Strategies that combined differentiated/small group instruction with maximizing instructional 
time included using homeroom or lunch periods as “viable options for quick-hit interventions” for students 
struggling in particular areas, and staggering literacy blocks across classrooms so some students could get a 
double dose of a given skill in different classrooms. The materials explained that this practice should target a 
specific subset of students struggling to reach proficiency, so it would be a form of differentiating instruction 
across students while maximizing instructional time in a given subject. 


The professional development materials did not offer explicit guidance on how teachers or other school staff 
should go about selecting a specific instructional strategy for their situation. The implicit approach conveyed in the 
materials is that teachers should collaboratively decide which specific strategy they should select, given their 
specific data. The professional development sessions were structured to engage participants in collaborative 
activities. Although a consultant from Focused Schools led these sessions, the idea was that participants would 
go back to their schools and conduct the same activities, either in teacher collaboration team meetings or in more 
general professional development sessions for teachers at the school. This collaboration was expected to lead to 
the selection of appropriate evidence-based instructional practices to boost student achievement in a classroom 
or school. As part of the ongoing data cycle, after trying out these practices, the teachers would use later team 
meetings to examine student data and discuss whether they thought the approach was successful. 


Note 
1. Hattie, John. (2008). Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. 
Abingdon, United Kingdom: Routledge. 
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Table 1.1. Ongoing professional development schedule and topics 


SY -1-4-) (0) 1 a BY 1) PTX} (0) aM did (-Me- Tale Me) o) (ex-Me-Vo lol c-\-t-J-10| 


1 January Laying the Foundation for DDI 
2015 Brief DDI overview for grade-level chairs 
Best practices to promote working group productivity 
Setting up data walls in classrooms, public spaces, and in the data/resource room 
Collaborative cycle for data analysis 
Analyzing student data to assess core instruction and identify an instructional focus 


Activities: Discussing how data-driven school leadership and teacher collaboration teams 
can function successfully using focused meetings and clear roles; learning and applying 
a data protocol to analyze student data and identify an appropriate instructional focus for 
improvement efforts; and discussing the purpose and examples of data walls. 


2 February Looking at Student Data and Setting Goals 
2015 Key activities of the ILT 

Understanding summative, interim, and formative assessments 
Using evidence to identify effective instructional practices for different situations 
Activities: Discussing different types of assessments; reviewing descriptions of school 
leadership teams to identify strengths and weaknesses; learning a process for using 
data to identify standards and select instructional practices to support them; reviewing 
John Hattie’s research and completing an exercise on planning for implementation of 
evidence-based teaching practices (e.g., modeled writing, reciprocal teaching, mini- 


lessons). 
3 March Using Best Practices in Teachers’ Classrooms 
2015 Moving from identifying to implementing evidence-based instructional practices 


Looking at student work to drive instruction 
Handling conflict within teacher collaboration teams 


Activities: Answering and discussing questions about how teams move from identifying 
to implementing evidence-based practices to strengthen core instruction; reviewing 
strategies for dealing with conflict in teams; and practicing how to use a protocol for 
looking at student work as data to make instructional decisions. 


4 April School Walkthroughs 
2015 Assessing implementation of key DDI activities: setting goals, instructional focus, best 
instructional strategies, ILT functioning, effective analysis of data 
Using school walkthroughs to provide effective feedback and coaching to teachers 
Activities: Reviewing walkthroughs (reflective walkthroughs, peer coaching, learning 
walk, schoolwide walkthrough); team planning time to create detailed short- and long- 
term plans to ensure work continued for the remainder of the school year and into the 


fall. 
5 September Refresher/Orientation, Goal Setting, and Planning Professional Development 
2015 Brief DDI overview for new ILT members 


Plans for professional development on evidence-based instructional practices 
Reviewing new end-of-year data to set new grade-level student achievement goals 
Using grade-level goals as lead-in to student goal setting 


Activities: Completing a three-month professional development planner showing how the 
school will build teacher capacity around selected evidence-based practices; analyzing 
early data to determine SMARTe (Specific, Measurable, Attainable and challenging, 
Relevant, Time bound, and touch Every student) goals for the school and grade levels. 
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Table I.1. (continued) 


SY -¥-4-) (0) 1 a BY 1) Session title and topics addressed 
6 October Sustaining the Work 
2015 Reviewing professional development plan to build expertise on DDI 


Ensuring use of evidence-based instructional best practices 
Student goal setting 
Planning for the sustainability of DDI 


Activities: Working with peers to share progress and opportunities for growth with 
emphasis on their roles as data-driven leaders; discussing to sustainability. 


Source: Compiled by study team based on materials provided by Focused Schools. 


Technical assistance. Focused Schools consultants also provided customized technical 
assistance to data coaches, principals, and other instructional leadership team members. The 
consultants were expected to visit each school eight times between winter 2015 and spring 2016 
to provide assistance in person. In addition, Focused Schools offered customized technical 
assistance to data coaches and school staff through monthly phone calls, and additional phone 
calls or e-mails as needed. Finally, data coaches could receive additional technical assistance and 
support through a web-based SharePoint site, designed to facilitate information sharing among 
the data coaches and consultants. The SharePoint site featured an online discussion board 
designed to facilitate discussions among the data coaches across all treatment schools. 


Data Coaches and Their Roles in Treatment Schools 


With input and support from Focused Schools and the study team, each district hired data 
coaches to work with fourth- and fifth-grade teachers in treatment schools. Focused Schools was 
involved in screening candidates in most districts, but ultimately district staff made the hiring 
decisions. To be qualified, individuals did not need previous experience as a data coach but were 
supposed to have experience working with student data and be experienced educators. It was 
hoped the original hires would stay throughout the two years. Each data coach worked half-time 
with a given school and thus could work with one or two schools in a district. As leaders of DDI 
implementation, data coaches were charged with conducting the following types of activities. 


Work in partnership with school principals. Data coaches were expected to help the 
principal lead DDI activities and establish a culture of data use in their schools. It was assumed 
that if the initiative were to succeed, it would depend substantially on principals taking a leading 
role in supporting their teachers and generally guiding the DDI activities in their schools. 
Coaches were supposed to meet weekly with principals. 


Set up and help lead an instructional leadership team in each school. This team, which 
included the data coach, principal, and other school leaders, was charged with guiding the 
school’s adoption of DDI. They were expected to meet once to twice a month, and their work 
was to analyze schoolwide data to formulate student achievement goals, identify an instructional 
focus for improvement efforts that would address identified weaknesses in the school’s 
instruction in grades 4 and 5, select evidence-based promising instructional strategies (best 
practices) to address the instructional focus they had selected, and monitor students’ progress 
toward the achievement goals. 


Set up and lead two teacher collaboration teams in each school. These teacher 
collaboration teams—one each for teachers in fourth and fifth grades—were supposed to operate 
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like professional learning communities, in which teachers would work collaboratively to 
implement DDI. The teams were charged with undertaking structured activities to analyze and 
interpret student data for their grade and individual classes; setting learning goals for their 
students based on this data analysis; identifying and implementing instructional strategies to help 
students meet the learning goals; and monitoring students’ progress toward the goals. By 
working collaboratively in these teams, the hope was that a teacher struggling with a particular 
topic or subset of students could identify best practices from their teaching colleagues who had 
been more successful in similar situations, as suggested by Hamilton et al. (2009). All teachers in 
the target grade levels were supposed to participate in their respective teacher collaboration team. 
Teams were expected to meet once or twice a month. 


Provide direct support to teachers. In addition to their work in leading the teacher 
collaboration teams, data coaches were charged with providing other supports to teachers 
working to implement DDI strategies. They were expected to meet one on one with teachers as 
needed to help them interpret student data and implement evidence-based instructional strategies, 
and also to provide feedback on their progress. They could also conduct school walkthroughs 
and classroom observations. Finally, data coaches were expected to support teachers’ efforts by 
providing physical resources such as (1) a professional development area or resource room 
containing DDI materials and (2) data walls, which are visual displays of student data posted in 
classrooms or workrooms where teachers could track the progress of students relative to 
benchmarks. The data walls were intended to support teachers’ efforts to use data to improve 
their instruction and boost student achievement.? 


Research Questions 
To describe the implementation and impacts of the support for DDI intervention, this study 
addressed four research questions: 


1. To what extent was support for DDI implemented as planned in treatment schools? 


2. What differences existed between treatment and control schools in key DDI-related 
supports, services, and activities? 


3. How did support for DDI affect teachers’ and principals’ use of data and teachers’ 
instructional strategies? 


4. How did support for DDI affect students’ achievement? 


3 Coaches created and encouraged teachers to create data walls in public spaces to identify school-wide achievement 
goals and celebrate and build momentum towards those goals. Data walls in non-public spaces (such as the resource 
room in which teacher teams met) were intended to highlight differences in student achievement across classrooms 
to allow the team to identify strengths and weaknesses in the teaching practices of individual members, and 
encourage teachers to discuss their different teaching approaches. Data walls were also meant to help teachers group 
students for differentiated instruction and show how students were progressing over time. These walls were updated 
after interim assessments to help students see their growth, and teachers were encouraged to maintain the 
confidentiality of students when displaying data in a public manner. 
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Chapter II of the report describes our approach to addressing these questions. Chapter III 
describes the results of our implementation analysis. We used data collected primarily from data 
coaches to address the first research question and data from surveys of teachers and principals in 
study schools to address the second research question. Chapter IV describes the results of our 
impact analysis, in which we used data from the teacher and principal surveys to estimate the 
impacts of DDI on how teachers and principals used data and which instructional strategies 
teachers implemented (research question 3), and district administrative records to measure 
student achievement and estimate the impacts of DDI on student achievement in math and 
English/language arts (research question 4). 
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li. STUDY SAMPLE, DESIGN, DATA, AND METHODS 


This chapter gives an overview of the key components of the evaluation—our study sample, 
experimental design, data collection, and analytic approach. Appendix A provides additional 
details. 


Study Sample 


The sample for the study consists of principals, teachers, and students within a set of schools 
and districts. We first describe how we selected the study districts and schools within those 
districts, and then characterize the samples of principals, teachers, and students in those schools 
from which we collected data. 


District and School Samples 


To implement the study of DDI, in spring/summer 2014, we recruited 12 districts that 
regularly administered interim and summative assessments, and regularly provided student 
assessment results and other types of data to teachers. We selected the districts and schools for 
this study purposefully, targeting those that currently did not have data-use initiatives in place 
but would be well suited to implement DDI. As a result, the study sample consists of districts 
and schools that otherwise might have been interested in implementing DDI, although it is not 
statistically representative of a broader group of districts or schools serving fourth- and fifth- 
graders nationally. 


Two key eligibility criteria guided the selection of districts in the spring and summer of 
2014. To effectively implement DDI, teachers and administrators in participating schools would 
need to be able to monitor student achievement on a regular basis using student data from both 
summative end-of-year and interim assessments (the latter are given to students periodically 
during the year to evaluate their learning progress). Thus, the first criterion was that districts had 
to plan to uniformly administer summative and interim assessments during the 2014-2015 and 
2015-2016 school years. These assessments are commonly administered and this requirement did 
not substantially reduce the pool of districts eligible for the study. 


The study relied on each district’s existing interim and summative assessments. Because the 
intervention was designed to help teachers use data from interim assessments to modify their 
instruction in order to improve students’ summative assessment scores, each participating district 
was required to administer reasonably well-aligned assessments, ideally by the same test 
developer. To meet this alignment goal, the study team targeted their recruiting efforts to states 
and districts in the Smarter Balanced Assessment Consortium. The tests developed by this 
consortium were designed to measure student achievement relative to Common Core standards, 
so we expected the consortium’s interim and summative assessments to be aligned with one 
another. When a district used assessments from different test publishers, we would consider 
including that district if both their interim and summative assessments were aligned to the same 
common set of standards (often state-defined standards). 


The second key criterion was that the district could not have an ongoing DDI intervention or 
plans to begin one within the intervention period, so we could ensure a clear difference between 
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the treatment and control schools regarding the presence of DDI. This approach would allow us 
to address the question of what impact DDI would have in a school that had not used this 
strategy before. 


Once we identified districts that met the above criteria, we identified a subset of eligible 
schools within them. The schools we selected met the following criteria: (1) they were willing to 
implement DDI and (2) they had no school-specific data-use initiatives in place. In addition, we 
wanted to ensure that a substantial proportion of the students served by study schools within a 
given district were low-income students, which we defined on the basis of eligibility for free or 
reduced-price meals. We did not require that schools meet a specific threshold for the proportion 
of low-income students, but in determining which schools within a district would be included in 
the study, we favored those with larger proportions. 


The final sample included 12 districts and 102 schools within those districts. The districts 
were medium sized and large, ranging from 9,409 to 52,834 students (with a median enrollment 
of 17,821 students per district). The districts were geographically diverse, located within eight 
states representing the East, South, Midwest, and West (figure II.1). 


Figure II.1. States represented by districts in study sample 


Created with mapchart.net © 


Note: Study districts were located in states shaded in red. 


Students in study schools were also economically disadvantaged and demographically 
diverse. Nearly two-thirds of students (64 percent) in study schools were low income—eligible 
for free or reduced-price school meals—and just over half of the students were White. The study 
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districts were similar to all districts in the United States in race, poverty, and the presence of 
English language learners. The study schools had a similar proportion of students who were 
White and races other than White relative to the national average; however, a larger percentage 
of students in the sample were low income. Appendix table A.1 includes additional information 
on students’ characteristics in study schools and districts. 


Principal, Teacher, and Student Samples 


The principal, teacher, and student samples were representative of the schools and grades 
included in the evaluation. The study team collected data from the full set of principals in all 102 
study schools as of spring 2016. For teachers, the initial random sample included 501 full-time 
fourth- and fifth-grade math and English/language arts teachers who taught in study schools 
during 2015-2016. The initial teacher rosters provided by the district included 546 teachers. 
From this frame, the team included teachers in study schools with four or fewer teachers in these 
grades in the study sample with certainty (n=138) and randomly selected teachers within each 
school that had five or more teachers, such that a minimum of four teachers would be included in 
the initial sample from each of these schools. After dropping teachers determined to be ineligible 
because they were not currently teaching at the school or did not teach the targeted grades or 
subjects, the final teacher sample size was 470. 


The student sample included all 12,535 fourth- and fifth-grade students in study schools as 
of spring 2016. Although we would not expect the DDI intervention to affect student mobility, 
and hence the composition of students in treatment schools, we conducted a sensitivity test using 
an alternative student sample (see appendix A for more detail). This sample included students 
enrolled in second and third grades in study schools as of spring 2014, before random 
assignment. 


Data Coach Sample 


All coaches who participated in the intervention were included in the sample of coaches. 
Coaches were asked to complete weekly logs about their coaching activities and to participate in 
one-on-one interviews during the fall and spring semesters. Coaches who were assigned to more 
than one school were asked to complete separate logs and interviews for each of the schools. 
The data coach sample included 39 coaches assigned to 50 schools in 2015-2016, including 11 
coaches who were assigned to two schools. At one school, the 2014-2015 data coach did not 
return for the 2015-2016 school year and was not replaced. 


Experimental Design to Estimate the Impact of Support for DDI 


To ensure that the study’s findings on the impacts of support for DDI can be attributed 
solely to the intervention described in Chapter I, we used an experimental design in which we 
randomly assigned schools within each study district to treatment and control groups. Schools in 
the treatment group received the half-time data coach and DDI professional development and 
technical assistance from Focused Schools, and implemented DDI. Schools in the control group 
proceeded with business as usual. We then followed each group of schools over a year-and-a- 
half follow-up period and compared outcomes measured at the end of this period. 
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Approach to Random Assignment 


We randomly assigned schools rather than individual students because we implemented a 
school-level intervention (in fourth and fifth grades). To improve the precision of the impact 
estimates, we used a matched pair design (Imai et al. 2009). We first identified pairs of schools 
within each district that had similar prior test scores and demographic and socioeconomic 
characteristics. We then randomly assigned one school within each pair to the treatment group 
and the other to the control group. Overall, we formed 51 matched pairs of schools in the 12 
study districts, resulting in 51 treatment and 51 control schools.* The study design gives it 
enough statistical power to detect effects on student outcomes that are 0.12 standard deviations 
(SD) or larger and effects on teacher and principal outcomes of 0.33 SD and 0.59 SD, 
respectively. (See appendix A for more detail.) 


Baseline Characteristics of Treatment and Control Schools 


An important consideration for this matched pair experimental design is that treatment and 
control schools needed to be similar at the beginning of the study; therefore, the only systematic 
difference was that schools in the treatment group received the DDI intervention, whereas those 
in the control group did not. By starting from the same point, we could attribute any later 
differences in outcomes between the two groups to the impacts of support for DDI. 


As expected, the characteristics of schools, students, and teachers in the treatment group 
were similar to those in the control group in most dimensions. Figure II.2 shows selected 
baseline characteristics of students and teachers assigned to schools in the treatment and control 
groups. Additional comparisons are presented in appendix A. There were no statistically 
significant differences between the two groups in the schools’ fourth- and fifth-grade 
achievement scores, or in most of the other characteristics of students and teachers. For example, 
teachers in treatment and control schools had similar amounts of prior teaching experience and 
similar educational credentials. We did find differences in students’ racial/ethnic composition, 
with students in treatment schools less likely than those in control schools to be White and 
Hispanic, and more likely to be Black (appendix table A.3). We also found that students in 
treatment schools were less likely to be English language learners. These differences did not 
translate into a significant difference in baseline student achievement, however. 


4 As described in appendix A, we initially randomly assigned 104 schools in 52 pairs, but later dropped one pair 
from the study sample. The school assigned to the treatment group within this pair received the DDI intervention for 
the full evaluation period. However, during this period (between the 2014-2015 and 2015-2016 school years) the 
school assigned to the control group began serving only kindergarten through second grades. Because this school no 
longer served the grades that were the focus of the intervention—fourth and fifth—we dropped the pair from the 
study sample. 
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Figure II.2. Baseline characteristics of treatment and control students and 
teachers 
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Source: District student records, teacher survey, 2012-2013 Common Core of Data, State Departments of 
Education and websites. 


Note: N = 12,036 students, 421-435 teachers, 102 schools. 
*Impact is statistically significant at the .05 level, two-tailed test. 


Data Sources 


Three types of data formed the basis of the analysis for the study. First, we collected 
information from data coaches as part of the study team’s monitoring of DDI implementation in 
treatment schools. Second, we conducted surveys of the principals and teachers in both treatment 
and control schools to describe the implementation further and measure key intermediate 
outcomes. Third, we obtained student-level administrative data from participating districts to 
measure student characteristics and achievement. Table II.1 summarizes these data sources and 
presents response rates. 
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Table II.1. Data sources for the DDI study 


ISX =X-) oe) a1} 
Data source F-Taa) ) (-) Type of information re} Wy) 


Program monitoring data 
Data coach at each treatment 


Stas . = 
Coach logs school Key DDI activities during week at school 88 
Fall 2015 data Data coach at each treatment Summary of DDI implementation in the 90 
coach interview school school 
Spring 2016 data Data coach at each treatment Summary of DDI implementation in the 92 
coach interview school school 
Principal and teacher data 

er Principal at each study school in Professional development and data-related 
PHneipel SHNey spring 2016 activities in schools = 


Professional development, data-related 


Sample of fourth- and fifth-grade activities in schools, and teacher 93 


Teacher survey teachers at each study school in collaboration during common planning 
spring 2016 : 
times at school 
Student data 
District Fourth- and fifth-grade students Student characteristics and performance on 
administrative at each study school in spring state assessments, 2013-2014 through 100° 
records 2016 2015-2016 
Note: Survey response rates are shown for treatment and control groups combined (where appropriate). None of 


the differences in response rates between the treatment and control groups were statistically significant. 


4We obtained at least some completed coach logs from each of the data coaches, representing their work at all 
treatment schools. During the 2015-2016 school year, the coach in the average treatment school completed 31.7 
weekly logs, or 88 percent of the number that would be completed if a coach completed a log for each week of a 36- 
week school year. 


> This is a school-level response rate, indicating that we received administrative records on fourth- and fifth-grade 
students for each study school as of spring 2016. Information on the percentage of students enrolled in spring 2016 
for whom we received district administrative records on their test scores is presented in Appendix Table A.4, and 
indicates that we received scores for 96 percent of enrolled students. 


Two types of program monitoring data allowed us to describe the implementation of the 
DDI intervention and assess whether intervention activities took place as expected. The data 
coaches working at each treatment school were asked to complete weekly work logs to describe 
the DDI intervention activities they completed that week, as well as summarize the overall status 
of implementation at the school at that time.° In addition, the study team interviewed the data 
coaches in fall 2015 and spring 2016. In these interviews, the data coaches reported on the status 
of implementation activities in their schools. Response rates on the data coach interviews were 
90 percent in fall 2015 and 92 percent in spring 2016. 


Surveys of school principals and the fourth- and fifth-grade teachers in both treatment and 
control schools provided information about data-related professional development and activities 
in schools. We conducted these surveys in spring 2016 and used the data to describe 


Although different data coaches completed different numbers of logs, we obtained completed logs for most of the 
weeks during the 2015-2016 school year for a majority of schools in the treatment group. In particular, the coaches 
at 35 percent of schools completed 36 or more weekly logs, 80 percent completed at least 27 logs, and 94 percent 
completed 18 or more logs. 
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implementation of DDI, and whether and how it led to differences in data-related activities 
between schools in the treatment and control groups. We also used data from the principal and 
teacher surveys to collect background information about educators, and measure intermediate 
outcomes that the logic model indicates DDI should affect, including principals’ and teachers’ 
data use, and teachers’ instructional practices. The response rates of 95 percent on the principal 
survey and 93 percent on the teacher survey did not differ significantly between the treatment 
and control groups. 


Finally, we collected data from study district administrative records on fourth and fifth grade 
students enrolled in treatment and control schools in spring 2016. The data included information 
on students’ characteristics and their scores on state assessments in math and English/language 
arts, covering the 2013-2014 through 2015-2016 school years. We used the data to estimate the 
impact of DDI on student achievement, as well as describe the student sample and compare 
characteristics between students in treatment and control schools. 


Analytic Approach 


The study included both an implementation and an impact analysis. We designed the 
implementation analysis to describe the DDI activities implemented in treatment schools and 
characterize the resulting differences between treatment and control schools in their data-related 
activities. The analysis relied on data from the data coach logs and interviews as well as the 
principal and teacher surveys. The analytic approach was primarily descriptive. 


The impact analysis examined the support for DDI intervention effects on a set of 
intermediate outcomes, along with the ultimate outcome of interest—student achievement. We 
first estimated impacts on intermediate outcomes reflecting principal and teacher practices that 
DDI was expected to influence according to the logic model. The logic model posited that 
changes in the intermediate outcomes, in turn, would lead to improvements in student 
achievement. 


The study’s random assignment design allowed us to estimate the impacts of support for 
DDI by measuring differences in outcomes between principals, teachers, and students in 
treatment and control schools. We used a linear regression model to compare outcomes of the 
two groups while accounting for the random assignment design, including the assignment of 
schools rather than students and the pairing of schools that were similar before random 
assignment. For student outcomes, the model included covariates to improve the precision of the 
estimates and account for any baseline differences between students in treatment and control 
schools. These covariates also improved the precision of the student-level model. For principal 
and teacher outcomes, the model did not include covariates other than matched-pairs indicators.° 


® With treatment status and matched-pair indicators included in the model, there were limited degrees of freedom 
remaining for any additional covariates in the principal model because there was only one observation per school. 
For the teacher model, we ran analyses on 15 key outcomes both with and without covariates. For teachers, the 
model with covariates included race, education level, certification type, sex, lead teacher role, and years of teaching 
experience. We found that including these additional covariates did not explain much of the variation in teacher 
outcomes, with an average increase in the adjusted r-square of .01 across all models. We also found that the 
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Thus, the model’s impact estimates reflect simple treatment-control differences. For each model, 
we used sampling weights that gave equal weight to each study school, ensuring that the 
estimates reflected the impact of DDI in the average study school. When testing the statistical 
significance of estimated impacts on teacher and student outcomes, we calculated standard errors 
and p-values that account for the clustering of students and teachers within schools. Appendix A 
provides more details on analysis methods. 


We also examined the impacts of support for DDI separately for selected subgroups of 
students or schools. This approach allowed us to measure whether the DDI intervention was 
more effective for some groups of students or schools than others. For example, because the 
intervention was implemented in fourth and fifth grades over a year and a half (December 2014 
through the 2015-2016 school year), the students in the two grades could have been influenced 
over different periods of time. In particular, the fifth-grade student cohort—students in fifth 
grade in 2015-2016—had the possibility of being directly affected by DDI over one-and-a-half 
school years, whereas the fourth-grade cohort could have been affected only during a single 
school year. Because of the longer possible exposure to DDI among fifth-graders, we estimated 
impacts separately by grade. 


We also estimated impacts separately by students’ baseline levels of achievement. The 
nature of the DDI intervention suggests that teachers may revise their instructional practices 
differently for different groups of students in their classrooms. For example, they may provide 
additional one-on-one instruction for students struggling to master certain topics. Thus, we 
hypothesized that the impact of DDI may have been different for students at different baseline 
achievement levels. 


Among schools, we examined just one set of subgroups. We hypothesized that because the 
DDI intervention was complex, it would be implemented more effectively in those schools more 
ready to do so at the beginning of the implementation period. For instance, schools that already 
had instructional leadership teams and grade-level teacher collaboration teams that were using 
student data were defined as being more ready to implement DDI. These schools may have been 
better positioned to immediately support teachers in their efforts to analyze data and improve 
their instructions than schools that did not have the those structures in place. If so, DDI may have 
led to greater improvements in student achievement in those schools with high levels of 
implementation readiness. Thus, we estimated impacts separately for schools with higher versus 
lower readiness levels, with each matched pair of schools classified into a readiness category 
based on our assessment of implementation readiness in the treatment school in the pair (since 
we did not have implementation readiness information for the control school). 


estimated treatment effect was similar with or without the additional control variables, and that including the 
covariates did not increase the precision of our measurement of the effect, as the standard errors of the treatment 
coefficient slightly increased on average in the expanded model. Therefore, we present results from the simple 
model including only treatment status and matched pairs. 


NCEE 2019-4008 Evaluation of Support for Using Student Data to Inform Teachers’ Instruction 21 


il. IMPLEMENTATION OF THE PROFESSIONAL DEVELOPMENT PROGRAM TO 
SUPPORT DATA-DRIVEN INSTRUCTION INTERVENTION 


This chapter describes the implementation of the support for data-driven instruction (DDI) 
intervention this study has examined. The intervention aimed to build capacity for DDI in 
treatment schools through key supports, provider services, and school-level activities. Chapter I 
described in detail these three key components of the DDI intervention as it was designed. This 
chapter first describes how these key components of the intervention were implemented in the 
treatment schools, with a focus on fidelity to the intended model. Next, it describes how the 
intervention affected data-related activities, using data from the study’s treatment and control 
schools. It concludes with a summary of key implementation findings. 


The Intervention as Implemented 


In examining the implementation of the DDI intervention in treatment schools, we focused on 
the three key components: (1) supports provided to the treatment schools as part of the DDI 
intervention, including data coaches assigned to the schools and the intervention provider; (2) 
professional development and technical assistance services from the provider; and (3) school- 
level activities expected to occur as part of a treatment school’s study participation. Together, 
these three components were expected to build school capacity for implementing DDI in the 
treatment schools, which in turn was expected to increase the use of data by teachers and school 
leaders, improve teachers’ instructional strategies, and ultimately improve student achievement, 
as illustrated by the logic model presented in Chapter I. Focused Schools, an experienced 
provider of DDI services, was the external service provider responsible for providing and 
guiding implementation of intervention-related services in the treatment schools. In this section, 
we report on the extent to which treatment schools were successful in implementing key 
intervention components. 


Implementation of Intervention Supports 


Intervention supports were provided as planned to the vast majority of study schools 
and districts throughout the study period. Two key supports were provided to treatment 
schools as part of the study: (1) professional development and technical assistance consultants, 
and (2) data coaches. Eight consultants from Focused Schools provided services in the 12 study 
districts. All consultants were former principals with experience in data-driven instruction and 
school turnaround initiatives. Each treatment school was provided with a half-time data coach to 
work primarily with teachers in grades 4 and 5. Data coaches were the key support that made the 
DDI intervention possible in treatment schools. Each study district identified, screened, and hired 
them, with input and support from the study team and Focused Schools. 


During the 2014-15 school year, the districts hired data coaches who worked in all 51 of the 
study’s treatment schools. In 2015-16, half-time data coaches were present in 50 of the 51 
schools. In one school, the original coach left in summer 2015, and the district was not able to 
hire a replacement. Most schools had the same data coach in both study years, as well as the 
same principal (appendix figure B.1). Data coaches, on average, were experienced educators, but 
most did not have previous data coach experience. Such experience was not a required 
qualification but could have been related to their effectiveness as data coaches. All 2015-16 data 
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coaches were former teachers, but only 36 percent had previous coaching experience and few 
had worked as a data coach previously. Nevertheless, coaches reported having more than a dozen 
years of experience working with student data, on average, regardless of their previous roles 
(appendix table B.1). 


Implementation of Provider Services 


The DDI support provider, Focused Schools, delivered support to schools through 
professional development to each district as planned, in most respects. As detailed in 
Chapter 1, professional development provided by Focused Schools included an up-front 
orientation to DDI in December 2014 and additional training sessions during 2015. Consultants 
from Focused Schools held six professional development sessions in each study district for 
coaches and school leaders throughout the study period, as planned (table I.1). It was expected 
that data coaches and principals would attend all of the sessions offered by the consultants. 


Although Focused Schools conducted all required professional development sessions, 
participant attendance at these sessions from the treatment schools was high but not perfect, 
especially for principals. Principals’ participation decreased over time, from 89 percent at the 
first session to 75 percent at the sixth and final session (appendix table B.2).’ Data coach 
participation was higher than that of principals but also lower than expected by the intervention 
design, ranging from 87 to 97 percent across the six sessions. Nevertheless, most coaches 
reported some benefits from the professional development and technical assistance provided as 
part of the intervention, with more than three-quarters of them (78 percent) reporting that their 
training had prepared them to carry out most or all of their tasks by spring 2016. Still, about one- 
fifth (22 percent) reported that their training either did not prepare them or prepared them to 
carry out some but not all tasks by the end of the study period (appendix table B.3).® 


Focused Schools consultants conducted most, but not all, of the expected technical 
assistance visits to treatment schools. The consultants were expected to visit each school eight 
times between January 2015 and March 2016, with visits coinciding, when possible, with 
professional development sessions. These visits provided an opportunity to check on progress, 
offer advice, and answer any questions. On average, treatment schools received 6.6 visits from 


2 Despite principals’ attendance at PD sessions being lower than teachers, principals who attended these sessions 
gave the sessions a slightly higher average rating on the quality of the sessions than other participants. The average 
rating from principals was 4.77 on a scale of | to 5, compared with an overall average rating among all participants 
of 4.72. 


Among coaches who reported that the training did not prepare them to carry out all tasks, some provided further 
explanation. The issues they cited were varied, although one theme was that the training was not specific enough. 
For example, one coach said that the “DDI expectations feel vague.” Others reported sometimes being “a little 
confused about the purpose of different activities” or that the training should provide “a more developed 
understanding of what DDI is.” Another issue raised by coaches involved the timing of the material covered in the 
training—that it was not aligned with what was going on in their school at the time. 
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their consultants during the study period—fewer than the expected 8. However, most schools (76 
percent) either met the target of eight visits or fell one or two visits short.” 


Although Focused Schools delivered technical assistance primarily during regularly 
scheduled school visits, other forms of help were also available to data coaches and other staff in 
treatment schools. For example, consultants were expected to communicate at least monthly with 
data coaches, either through email or phone. In addition, data coaches and consultants could 
share information using a web-based SharePoint site hosted by the study team. For the most part, 
consultants communicated with data coaches as expected. Nearly all data coaches (94 percent) 
reported at least monthly email or phone contacts with their consultant. 


Implementation of School-Level Activities 


Within the treatment schools, each school’s leaders, data coaches, and teachers were 
expected to engage in the intervention through a variety of activities (see Chapter I). Principals 
were expected to provide overall leadership for data use, set expectations for teachers, and work 
consistently with the data coach. Two different types of working groups were to be formed in the 
treatment schools as part of building school capacity to analyze and use data effectively: (1) an 
instructional leadership team and (2) fourth- and fifth-grade teacher collaboration teams. In 
addition, data coaches were expected to analyze student data and support teachers in their DDI 
work through activities such as one-on-one meetings and classroom observations. As described 
below, most of these school-level activities were implemented as expected in the majority of 
treatment schools, although some schools did not implement all expected activities early enough 
to be in place for the full 2015-16 school year. 


Most treatment school principals were actively engaged in the intervention, providing 
overall leadership for DDI and supporting their data coaches. As their schools’ main 
instructional leader, principals were expected to play a crucial role in establishing DDI 
implementation and data use more generally as a priority. They were expected to meet one on 
one with the data coach each week to strategize for DDI implementation, allocate time and 
resources to support DDI, and set expectations and provide guidance on data use by teachers. 
Most principals met regularly with data coaches, although they typically did not fully meet the 
expectation of weekly meetings. A majority (59 percent) met at least three times a month, 
whereas only 8 percent met less than twice a month (appendix figure B.2). Data coaches 
generally viewed principals as being supportive, with 72 percent giving the principals the highest 
score on a five-point scale rating their support (appendix table B.4). 


Most principals also set expectations and provided guidance on data use by teachers as 
expected. Principals had discretion over how to provide this guidance, and how often (that is, 
Focused Schools did not specify how principals should provide guidance). Between two-thirds 
and 90 percent of principals reported providing guidance to teachers at least monthly or several 


° School visits were recorded in consultant logs, which did not provide information on the reasons a school did not 
receive a visit. It is possible consultants occasionally forgot to record a school visit, visited with the coach of two 
schools at only one school site, and/or encountered difficulties when scheduling visits with coaches. 
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times per term on a variety of data use topics (figure II.1)'°. For example, principals commonly 
provided guidance to teachers on the types of student data to examine (79 percent), how often 
they should examine these data (85 percent), and what evidence-based instructional strategies to 
use (89 percent). 


Figure II1.1. Percentage of principals in treatment schools providing guidance 
to teachers on aspects of data-driven instruction 


Percentage of principals who reported providing guidance to teachers at least 
monthly or several times per term on... 


Protocols or strategies to analyze student data 66% 


Achievementor proficiency improvement goals 715% 


Types of student data to examine 79% 


How often to examine student data 85% 


Evidence-based instructional strategies to use 89% 


Source: Principal survey (n = 92-94). 


Most treatment schools had actively involved school leaders through an instructional 
leadership team. Each treatment school was expected to establish an instructional leadership 
team composed of the principal, data coach, grade 4 and 5 chairs, and any other key school 
leaders seen as important for establishing leadership and a culture of data use. In most schools, 
these teams had been established and met the intervention target of meeting once or twice per 
month during the 2015-16 school year. Nearly all treatment schools (91 percent, according to 
data coaches) had established instructional leadership teams by fall 2015, the expected time 
frame. In fact, data coaches at more than half the schools (56 percent) indicated that a team 
similar to an instructional leadership team predated the intervention. However, they also noted 
that the composition of the team and the focus of their work or frequency of meetings were 
modified to meet the expectations of the intervention—for example, 95 percent of schools with a 
team reported a greater emphasis on data use as a result of the intervention. 


10 While this section focuses on principals of treatment schools, comparisons between treatment and control school 
principals are provided in appendix B. Principals in control schools were less likely than those in treatment schools 
to provide guidance to teachers about data-driven instruction (appendix figure B.4). 
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Instructional leadership teams in most treatment schools had engaged in the types of 
activities expected of them under the support for DDI intervention, although some were 
delayed in doing so. Instructional leadership teams were expected to select an instructional 
focus, schoolwide improvement goals, and best practices as part of the DDI intervention. In most 
treatment schools, instructional leadership teams completed these activities. The teams in a 
majority of schools met each of these DDI expectations in both fall 2015 and spring 2016 (figure 
IH.2). However, the teams in a substantial minority of treatment schools completed these 
activities well into the intervention period. In fall 2015, for example, more than a quarter of 
schools had failed to select an instructional focus, and somewhat larger percentages had failed to 
set schoolwide improvement goals and best practices. As a result, the full implementation was 
not in place in these treatment schools during the full 2015-16 school year. The study team 
conducted the fall interviews with data coaches in late September and early October, so some of 
the schools that had not met expectations by the interview date may have done so later in the fall. 
By spring 2016, instructional leadership teams in 93 percent of treatment schools had selected an 
instructional focus.'' Some coaches reported challenges in working with their schools’ 
instructional leadership teams (appendix table B.6), such as consistency in member participation 
and attendance (38 percent), following through on decisions (38 percent), and balancing DDI 
work with other school priorities and responsibilities (36 percent). These challenges, reported in 
spring 2016, may have contributed to a lack of implementation progress in some treatment 
schools. 


Based on the logic of the support for DDI intervention, instructional leadership teams were 
not told what instructional focus, schoolwide improvement goals, or best practices they should 
put in place. Instead, the teams should have selected them on the basis of the situation in their 
school, and in particular, on the basis of student data. Box 3 provides examples of treatment 
schools’ choices of an instructional focus and best practices. Most schools (70 percent) selected 
an instructional focus in the area of English/language arts (figure III.3).!* 


'l The instructional leadership teams at some schools had not met specific expectations of the intervention by spring 
2016. In particular, 7 percent had not selected an instructional focus, 14 percent had not selected schoolwide 
improvement goals, and 24 percent had not selected instructional best practices at the school level (figure III.2). 


!2 This estimate includes in the category of English/language arts schools that selected an instructional focus in 
reading, reading and writing, reading and math, writing, and vocabulary. 
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Figure III.2. Percentage of schools in which instructional leadership teams 
had completed key components of the data-driven instruction intervention, 
fall 2015* and spring 2016** 


Selected Instructional Focus 


Selected Schoolwide Improvement Goals 57] 


Selected Instructional Best Practices 


*First set of data bubbles; colored blue 
**Second set of data bubbles; colored red 
Source: Data coach interviews (n = 45-46). 


Note: Blue dots with no shadow represent the percentage of schools that had completed the intervention 
component by the fall 2015 interview; red dots with a shadow represent the percentage that had completed 
the component by the spring 2016 interview. 
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Box 3. What kinds of instructional focus and best practices did instructional 
leadership teams choose? 


Among treatment schools, 68 percent selected an instructional focus related to the content area of 
English/language arts (figure III.3), including those that selected a focus in vocabulary. According to the data 
coaches, schools most commonly selected reading as their focus for instruction within the English/language arts 
area. Only 6 percent of treatment schools opted to focus exclusively on math, whereas 14 percent selected an 
instructional focus in both math and reading. Notably, 16 percent of schools selected an instructional focus not 
associated with a content area (for example, academic vocabulary, increasing student engagement, improving 
classroom discussion, differentiated instruction) and 10 percent had not selected any instructional focus. 


Most treatment schools with an instructional focus also selected at least one instructional best practice, such as 
using text-based evidence or guided reading. The leadership teams reported a variety of instructional best 
practices across the schools (appendix table B.5). Many of the best practices identified were content related and 
aligned with a school’s instructional focus (for example, selecting math mindsets pedagogy as a best practice with 
a math instructional focus), but not always. Some best practices were not content-specific but rather more general, 
such as having reading as the instructional focus coupled with differentiated instruction and selecting appropriate 
goals for student achievement as the instructional best practices. 


Figure II1.3. Percentage of schools that selected an instructional focus in different areas, 
2015-16 


English/language arts 


@ Reading and Math 


= Math 
Other 
CNone 
Instructional 
Focus 
Includes Instructional Focus Includes 
Math 


English/language Arts 


Source: Data coach interviews for 50 treatment schools. 


Most treatment schools established teacher collaboration teams in grades 4 and 5 that 
met at least twice per month and engaged in some data-related activities. Each treatment 
school was expected to establish two teacher collaboration teams—one for grade 4 and one for 
grade 5—consisting of the data coach and all teachers in that grade. Much of the on-the-ground 
work of DDI was expected to be done by these teams, as they were to meet at least every other 
week and act as professional learning communities, in which participating teachers work 
collaboratively under the leadership of the data coach. Nearly all treatment schools had 
established teams that met at least monthly by fall 2015, and two-thirds (65 percent in grade 4 
and 69 percent in grade 5) met at least twice a month (appendix figure B.3). Most treatment 
schools already had teacher teams in place before the intervention, but 88 percent of those with 
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preexisting teams had to make changes to their responsibilities or composition as part of the 
intervention, including a greater emphasis on data use. In working with teacher collaboration 
teams, data coaches reported challenges similar to but even more frequent than those they 
reported in working with instructional leadership teams. These challenges included follow 
through on meeting decisions (57 percent), balancing DDI work with other work and priorities 
(52 percent), consistent participation of members during team meetings (41 percent), and 
maintaining focus on data and instructional improvement (32 percent). 


Data coaches in most treatment schools regularly supported teachers in a variety of 
DDI activities outside of the instructional leadership team or teacher collaboration teams, 
as expected by the intervention. As part of their responsibilities, data coaches were expected to 
create data walls, conduct school walkthroughs, and support grade 4 and 5 teachers during one- 
on-one sessions and through informal classroom observations. Most coaches regularly engaged 
in these expected activities, with increases in the prevalence of the activities between fall 2015 
and spring 2016 (figure III.4). By spring 2016, for example, data coaches in all treatment schools 
met individually with teachers, about 80 percent conducted observations and walkthroughs, and 
nearly 90 percent had created data walls. Coaches reported data walls were located in hallways 
and classrooms. The most commonly and consistently displayed data were interim and formative 
assessment results, followed by summative assessment results and attendance data. !? 


Figure III.4. Percentage of schools in which data coach engaged in different 
activities, fall 2015* and spring 2016** 


Met Individually with Teachers 


Conducted Observations 


Conducted Walkthroughs 


Created Data Walls 


*First set of data bubbles; colored blue 

**Second set of data bubbles; colored red 

Source: Coach interviews, fall 2015 and spring 2016 (n = 45-46). 

< or > indicates that we have withheld the exact percentage to protect respondents’ confidentiality in accordance with 
National Center for Education Statistics statistical standards, but that the percentage is less than or greater than the 
number following the < or > symbol (U.S. Department of Education 2012). 


'3 Data coaches faced challenges related to data walls—40 percent of data coaches reported difficulties finding time 
to create or update the data walls. Overcoming teacher reservations about displaying data was also reported as a 
challenge by nearly one-third of the coaches. 
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Data coaches in treatment schools reported engaging in a variety of activities during 
teacher collaboration team meetings, not all of which were focused on using data to 
improve instruction. Student data guided many of the activities in which the teacher 
collaboration teams engaged, but information collected through the weekly data coach logs 
indicated that team activities differed substantially from school to school and across different 
periods of the school year. Some teacher collaboration teams focused on analyzing student data 
and implications for instruction, whereas others focused on other activities such as curriculum 
planning, onboarding new teachers, or scheduling schoolwide testing. In other words, based on 
data coach logs, schools appeared to differ in the extent to which teacher collaboration teams 
analyzed data, discussed links between data and instruction, and identified evidence-based 
instructional practices to put into place in their classrooms. The data from the weekly coach logs 
were not sufficiently and consistently detailed for us to report on exactly how specific data- 
related activities of these teams varied across treatment schools. However, the next section 
presents evidence on teacher collaboration activities more generally in treatment and control 
schools. 


Contrast Between Treatment and Control Schools in Data-Related 
Supports and Activities 


To the extent that the support for DDI intervention was implemented as planned, it should 
have resulted in clear differences between the treatment and control schools in the presence and 
use of certain supports, practices, and activities in schools. In particular, the intervention should 
have led to more of three kinds of activities in treatment schools than in control schools: (1) 
principals and teachers receiving data-related professional development; (2) teachers receiving 
guidance and support on data use from coaches and school leaders; and (3) teachers collaborating 
on data-related activities. These differences were expected to be apparent during the 2015-16 
school year as a precursor to the intervention having an intended short-term effect on data use 
and teachers’ instructional strategies, as well as intended longer-term effects on student 
achievement. 


Data-Related Professional Development 


Focused Schools provided a key component of the intervention in the form of professional 
development on DDI. As described in the previous section, all principals and those teachers who 
were grade-level chairs in treatment schools were required to attend the professional 
development sessions, and other teachers in the school should have received this training 
indirectly (although it may have been in the form of informal support and guidance, either as part 
of the teacher teams or outside of collaboration time). Thus, principals and teachers in treatment 
schools were expected to receive larger amounts of data-related professional development than 
those in control schools. 


Principals in treatment schools received more professional development on data- 
related topics than those in control schools. All treatment school principals reported receiving 
formal professional development on data-related topics during 2015-16, compared with 73 
percent of control school principals (table HI.1). The amount of data-related professional 
development also differed, with treatment school principals reporting more than twice as many 
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days as control school principals of professional development focused on topics related to how to 
use and analyze data to inform instructional practices (5.6 versus 2.7 days). The data-related 
training received by treatment school principals would presumably have included the two days of 
formal DDI sessions but would also include any other data-related professional development the 
principals received. 


Table II1.1. Principal professional development during 2015-2016 


Receipt of Professional Development Treatment Control Difference p-value 
Received any professional development (percentages) 100.0 89.1 10.9* 0.02 
Total days of professional development over school year 12.5 9.9 2.6" 0.04 
(average) 

Received professional development focused on topics related 

to how to use and analyze data to inform instructional practices 100.0 73.3 26.7* 0.00 
(percentages) 

Days of professional development focused on topics related to 

how to use and analyze data to inform instructional practices 5.6 2.7 3.0* 0.00 
(average) 


Professional Development Topics (percentages who reported topic was a major focus) 
How to analyze or interpret various types of student data to 


understand student needs ont ae weld on 
How to use data to set individual learning goals for students 38.3 8.5 29.8* 0.00 
How to change instruction based on data 38.3 <6.7 >31.6* 0.00 
How to use student data to monitor student progress toward 43.5 10.9 32.6* 0.00 
meeting learning goals 

How to use evidence-based instructional strategies to help 44.7 10.6 34.0" 0.00 
students meet learning goals 

Number of Principals—Range?* 45-47 45-47 90-94 


Source: Principal survey. 


Notes: The difference between the treatment and control estimates may not equal the impact shown in the table 
because of rounding. 


@ Sample sizes are presented as a range based on the data available for each row in the table. 

* Difference is statistically significant at the .05 level, two-tailed test. 

< or > indicates that we have withheld the exact percentage to protect respondents’ confidentiality in accordance with 
National Center for Education Statistics statistical standards, but that the percentage is less than or greater than the 
number following the < or > symbol (U.S. Department of Education 2012). 


Teachers in treatment schools were more likely than those in control schools to receive 
professional development on how to use and analyze data to inform instructional practices. 
Teachers in treatment schools reported receiving an average of 4 more hours (12.6 versus 8.3 
hours) of formal professional development on data-related topics during the 2015-16 school year 
than did control school teachers (appendix table B.7). Teachers in treatment schools were more 
likely to report receiving professional development on four data-related topics (figure III.5). In 
particular, they were more likely to receive professional development on how to use various 
types of student data to understand student needs (29 versus 15 percent), set individual learning 
goals for students (29 versus 17 percent), change instruction based on data (22 versus 16 
percent), and monitor student progress toward meeting learning goals (33 versus 14 percent). 


In interpreting these results, it is worth noting that most teachers in treatment schools (those 
who were not grade-level chairs) did not participate in the formal Focused Schools professional 
development sessions. They may have received that training indirectly or informally from data 
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coaches or other participants, but would not necessarily have interpreted it as formal professional 
development when responding to the survey questions. This aspect of the intervention 
implementation plan may explain why fewer than half of teachers in treatment schools reported 
receiving professional development on the specific data-related topics shown in figure III.5. That 
is, 1f teachers had reported both the formal and informal data-related training and professional 
development they received, it is possible that those percentages would have been higher, 
especially among teachers in treatment schools. ! 


Figure 111.5. Teacher professional development (PD) during 2015-2016 
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*Difference is statistically significant at the .05 level, two-tailed test. 


ST explore the possibility that these differences were driven primarily by grade-level chairs, we also examined 
treatment and control differences in professional development separately for grade-level chairs versus other teachers 
(appendix table B.7). In general, we found similar patterns for both types of teachers. For instance, 93 percent of 
grade-level chairs in treatment schools had DDI-focused professional development, compared to 77 percent of 
grade-level chairs in control schools. Among other teachers, the percentages were 83 percent and 73 percent, 
respectively. The number of hours of DDI-focused professional development was approximately 4 hours higher in 
treatment schools for both types of teachers, although this difference was not statistically significant for grade-level 
chairs. Differences in the topics addressed in the DDI-focused professional development varied somewhat by 
teacher type. The treatment-control difference in how to analyze data was only statistically significant for other 
teachers (27 percent versus 15 percent), while the gap in using data to set goals was only statistically significant 
among grade-level chairs (48 percent versus 11 percent). Within the full sample, there was no statistically significant 
difference between treatment and control teachers in the topic of evidence-based instructional practices. However, 
grade-level chairs in treatment schools were statistically significantly more likely than those in control schools to 
have had professional development with this focus (38 percent versus 19 percent). 
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Guidance and Support From Coaches and School Leaders in Data Use Activities 


We expected that the support for DDI intervention would lead to more data-related guidance 
and support for teachers from school leaders and data coaches in treatment schools than control 
schools. As noted earlier, school leaders, including the principal and instructional leadership 
team, were expected to promote a culture of data use and support teachers in their efforts to use 
data to improve their instruction in spring 2016. The data coaches were expected to work with 
teachers, both one on one and in teacher collaboration team meetings, on various aspects of data 
use. This coaching was intended to be the primary way to provide teachers with information 
about how to change instructional practices to improve student achievement, and therefore was 
critical to the implementation of DDI. Below, we document the treatment-control difference in 
the presence of a data coach and examine differences in the extent to which teachers in treatment 
versus control schools received data-related guidance and support from coaches or other school 
leaders. 


Treatment schools were more likely than control schools to have a data coach. Nearly 
all treatment schools had a data coach/staff person available on site to help teachers use data to 
improve instruction, compared with 36 percent of control schools. The presence of a data coach 
in more than one-third of the control schools is one sign that many of these schools encouraged 
teachers’ data use to some extent. Below, we describe in greater detail the specific kinds and 
amount of data-related support that teachers received in treatment and control schools. 


Teachers in treatment schools received more data-related support and guidance from 
school leaders on various aspects of data use. The most intensive way teachers could receive 
support on data use was through one-on-one interaction with a coach or school leader. Larger 
percentages of teachers in treatment than control schools reported working one on one with a 
coach or school leader on various aspects of data use at least monthly or several times per term 
(figure III.6). The support for DDI intervention may have led to even more frequent one-on-one 
support in treatment schools, and as a result the additional support may have been more likely to 
affect teachers’ instructional practices and student achievement. Thus, we also examined the 
percentage of teachers who reported receiving these types of support at least weekly or several 
times per month. While smaller percentages of both groups reported this more frequent support, 
teachers in treatment schools were more likely than those in control schools to report frequently 
receiving each form of support, and the differences were larger (appendix figure B.5). 
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Figure III.6. Percentage of teachers who reported working one-on-one with a 
coach or school leader at least monthly on different aspects of data use 
during 2015-2016 
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Source: Teacher survey (n = 432-433). 
*Difference is statistically significant at the .05 level, two-tailed test. 


Most notably, teachers in treatment schools were more likely than those in control schools to 
receive support on analyzing and interpreting student data (69 versus 56 percent) and use data to 
set individualized learning goals for students (66 versus 49 percent). The difference in the 
percentage of teachers receiving support to change instruction based on data was not statistically 
significant, consistent with the similar rates of professional development on changing instruction 
based on the data noted above. 


A somewhat less intensive form of support to teachers involved guidance and feedback from 
school leaders on data use that was not necessarily one-on-one. Teachers in treatment schools 
were more likely than those in control schools to receive this sort of support from school leaders. 
A larger percentage of teachers in treatment than control schools reported receiving guidance and 
feedback from school leaders on student goals at least monthly or several times per term. !° For 
example, 77 percent of teachers in treatment schools reported receiving at least monthly 
guidance on strategies to use to analyze student data, compared with 61 percent of teachers in 
control schools (table III.2). Teachers in treatment schools were also at least 15 percentage points 
more likely to report receiving feedback on their data analysis and use of data to guide 
instruction (75 versus 55 percent), guidance on setting goals for students (77 versus 62 percent), 
and feedback on student progress toward meeting those goals (75 versus 57 percent). 


'S Tt is also the case that a larger percentage of teachers in treatment than control schools reported receiving this 
guidance at least weekly or several times per month (appendix table B.9). 
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Table II1.2. Percentage of teachers who reported guidance or feedback from 
school leaders on data-related topics at least monthly or several times per 
term during 2015-2016 


Treatment Control Difference p-value 


Guidance on: 


How often to examine student data 89.9 76.8 13.1* 0.00 
Types of student data to examine 85.5 76.8 8.6* 0.01 
Protocols or strategies to use to analyze student data 77.0 60.5 16.5* 0.00 
Achievement or proficiency improvement goals for students 76.5 61.5 15.0* 0.00 
Evidence-based instructional strategies (best practices) to use 77.0 63.1 13.9* 0.00 
Feedback on: 

Data analysis and use of data to guide instruction 74.6 54.9 19.7* 0.00 
a of evidence-based instructional strategies (best 70.7 57.0 13.7% 0.00 
Student progress toward meeting achievement of proficiency 74.6 57.1 17.5* 0.00 


improvement goals 
Number of teachers (range)? 218-219 214-215 432-434 


Source: Teacher survey. 


Notes: The difference between the treatment and control estimates may not equal the impact shown in the table 
because of rounding. 


aSample sizes are presented as a range based on the data available for each row in the table. 
*Treatment-control difference is statistically significant at the .05 level, two-tailed test. 


Instructional leadership teams were more likely to engage in a variety of data-related 
activities in treatment than control schools. Just as teachers reported receiving more guidance 
and support from school leaders in treatment than control schools, the treatment school principals 
were more likely to report that school leaders—through the instructional leadership teams—were 
engaged in the kinds of activities that would support their teachers’ data use. Among the 15 
different data-related activities examined, instructional leadership teams in treatment schools 
were more likely than those in control schools to engage in 14 of these activities at least monthly 
(appendix table B.8). For example, instructional leadership teams in treatment schools were more 
likely to do the following: set achievement or proficiency improvement goals (72 versus 47 
percent); identify evidence-based instructional strategies teachers should use (85 versus 51 
percent); provide guidance on the types of student data teachers should examine (83 versus 35 
percent); and develop or plan data-related professional development for teachers (83 versus 45 
percent). 


Teachers in treatment schools were more likely than those in control schools to analyze 
and document student progress for visual displays throughout their schools. While common 
in both groups of schools, teachers in treatment schools were more likely to develop and update 
analyses of student progress at least monthly or several times per term. For example, 72 percent 
of teachers in treatment schools reported developing or updating visual displays in hallways or 
grade-level common areas, compared with 61 percent in control schools (figure III.7). Teachers 
in treatment schools were also more likely to develop and update analyses of student progress for 
the following: visual displays in the classroom (84 versus 76 percent); individual students in 
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folders, binders, or portfolios (84 versus 75 percent); visual displays in teacher lounges or staff- 
only areas (28 versus 17 percent); visual displays for school lobbies or public areas (31 versus 20 
percent); and visual displays in libraries or school-wide common areas (36 versus 28 percent). 


Figure III.7. Percentage of teachers who developed and updated analyses of 
student progress at least monthly or several times per term during 2015-2016 
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Source: Teacher survey (n = 421-422). 
*Difference is statistically significant at the .05 level, two-tailed test. 
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Teacher Collaboration on Data-Related Activities 


The support for DDI intervention was designed to provide teachers with dedicated time to 
engage in collaborative activities with other teachers related to data use. Unless similar efforts 
took place in control schools, this design should have led to more collaboration among teachers 
in treatment than control schools on various aspects of data use. 


Teachers in treatment schools received more training and coaching on data-related 
topics during common planning time. About half of the teachers in treatment schools reported 
receiving at least monthly or several times per term training or coaching during common 
planning time on how to analyze and interpret student data (52 percent) and on evidence-based 
teaching strategies (47 percent) (figure III.8). A lower percentage of teachers in control schools 
(34 percent) reported receiving each of these types of training at least monthly. !° 


Figure III.8. Percentage of teachers who reported receiving training or 
coaching on data-related activities in collaboration with other teachers 
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Source: Teacher survey (n = 421-422). 
*Difference is statistically significant at the .05 level, two-tailed test. 


Teachers in treatment schools were no more likely than those in control schools to 
regularly engage in most data-related activities during common planning time. In treatment 
and control schools, teachers were similarly likely to report engaging in several data-related 


ay larger percentage of teachers in treatment than control schools reported receiving this type of coaching or 
training at least weekly or several times per month (appendix figure B.7). 
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common planning time activities at least monthly or several times per term.'’ These activities 
included the key aspects of DDI the intervention aimed to promote, including jointly analyzing 
data to understand student needs, setting common learning goals, examining student progress 
toward these goals, exchanging ideas on how to change instruction based on data, and jointly 
modifying lesson plans based on data (figure III.9). 


Figure II1.9. Percentage of teachers who reported engaging in data-related 
activities with other teachers during common planning periods at least 
monthly during 2015-2016 
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Note: None of the treatment control differences was statistically significant at the .05 level, two-tailed test. 


Levels of collaborative data-related activities were high, even in control schools. In both 
groups of schools, many teachers worked collaboratively on these activities. For example, 79 
percent of teachers in treatment schools and 73 percent of those in control schools jointly 
analyzed data during common planning time to understand student needs. Teachers also 
frequently reported using this time to discuss changing instruction based on data, setting and 
monitoring student goals, and jointly modifying lesson plans based on the data. For example, 

63 percent of those in treatment schools and 61 percent in control schools reported engaging in 
this last activity at least monthly (figure III.9). 


'7 Teachers in treatment and control schools were also similarly likely to report engaging in these activities at least 
weekly or several times per term, with one exception (appendix figure B.8). Teachers in treatment schools were 
more likely to report frequently jointly analyzing data to understand student needs (48 versus 36 percent). 
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Summary of Implementation Findings 


The professional development program to support the intervention aimed to build capacity 
for DDI in treatment schools through implementation of key supports, provider services, and 
school-level activities. For schools to fully implement the intervention, they first had to hire a 
half-time data coach and agree to work with a consultant from Focused Schools, the study’s 
technical assistance provider. Second, the Focused Schools consultant had to deliver professional 
development and technical assistance to the schools as planned. Third, school leaders and data 
coaches had to engage in a set of activities in the schools designed to support teachers in using 
student data to improve their instruction. In most respects, the key aspects of the intervention 
were implemented as planned. Nevertheless, little evidence suggests that the intervention led to 
increased teacher collaboration around analyzing data or using data to adjust instructional 
practices. 


The DDI intervention successfully provided the key supports and services of data coaching 
and professional development or technical assistance on data use. 


e All treatment schools hired half-time data coaches, and these schools were much more likely 
than control schools to have data coaches during the 2015-16 school year. The data coaches 
that treatment schools hired were experienced educators, although most did not have prior 
experience in being a data coach. 


e Focused Schools delivered professional development and technical assistance as expected in 
most respects. Planned professional development sessions were held, and attendance of data 
coaches and other school staff was high. This resulted in the principals and teachers in 
treatment schools receiving more data-related professional development than those in 
control schools. 


The intervention was successful in engaging school leaders in treatment schools, who 
performed the key DDI activities supporting teachers as planned, although some began 
implementing these activities later than others. 


e Most principals met with data coaches regularly to monitor and support implementation of 
the intervention, and data coaches perceived them as supportive of DDI. Broader groups of 
school leaders became involved in DDI through instructional leadership teams. In most 
treatment schools, these teams met regularly and engaged in key DDI activities, such as 
selecting an instructional focus. As a result, teachers in treatment schools were more likely 
than those in control schools to report receiving guidance and feedback from school leaders 
on data-related topics. 


e Most treatment schools established teacher collaboration teams that met regularly, but there 
is little evidence to suggest that the intervention led to greater teacher collaboration around 
analyzing data or using data to adjust instructional practices. Teachers in treatment and 
control schools reported a similar frequency of data-related activities during common 
planning time, such as jointly analyzing data or jointly adjusting lesson plans based on data. 
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IV. IMPACTS OF PROFESSIONAL DEVELOPMENT IN SUPPORT OF DATA- 
DRIVEN INSTRUCTION 


As described in the logic model (figure I.1), the support for data-driven instruction (DDI) 
intervention examined in this report provided schools with a data coach and professional 
development, and involved school leaders and teacher collaboration teams in supporting 
teachers’ data use. These additional resources and supports were intended to help teachers 
analyze and interpret student data in ways that would allow them to improve their instructional 
practices. Although the intervention did not specify exactly what practices teachers should use, 
the idea was that through teacher collaboration and with the support of trained data coaches and 
school leaders, teachers would be able to use data to determine the instructional practices that 
would most benefit their students. And by leading to improved instruction, the intervention 
aimed ultimately to boost student achievement. This chapter assesses the impacts of DDI on both 
the intermediate outcomes of teachers’ data use and instructional practices and the ultimate 
outcome of student achievement. 


Impacts of Support for DDI on Teachers’ Practices 


Schools adopt DDI in hopes that by making greater use of data, teachers will better 
understand students’ learning needs in at least two ways. First, the data should tell teachers 
which concepts students understand and which they do not. Second, the data should shed light on 
the characteristics of the students who are struggling the most. By pinpointing students’ learning 
needs, DDI should help teachers figure out how to adjust their instruction, and who to target with 
those adjustments, to better address these needs. As noted above, DDI does not specify exactly 
what instructional strategies teachers should use, but common instructional strategies 
emphasized in the Focused Schools professional development and in the DDI research literature 
(Hamilton et al. 2009) include increasing instructional time on particular topics and 
individualized or differentiated instruction. 


The goal of the DDI intervention was to support teachers in ways that made it easier for 
them to use data to improve their instruction. The data coaches, professional development 
provided by Focused Schools, involvement of school leaders, and use of teacher collaboration 
teams were the main ways that the intervention supported teachers. This section examines 
whether these supports were successful in increasing teachers’ data use and leading to changes in 
their instructional practices. Although neither increased data use nor revised instructional 
practices would guarantee improvements in student achievement, they are key intermediate 
outcomes based on the logic of DDI. 


Teachers’ Data Use 


Support for DDI had a limited impact on the extent to which grade 4 and 5 teachers 
reported using data to help guide their math and reading instruction. Despite the focus of 
DDI on teachers’ data use, the intervention did not increase teachers’ use of the data practices 
examined. Similar percentages of teachers in treatment and control schools reported using each 
of nine practices in each subject (figures [V.1 and IV.2). Among math teachers, for example, 50 
percent of those in treatment schools and 47 percent in control schools reported using data at 
least several times per week to understand student learning needs. Teachers in treatment and 
control schools were also similarly likely to use data for small group instruction (44 versus 45 
percent in math, 41 versus 38 percent in English/language arts) or to use data to adjust the time 
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they spend on different topics (44 versus 40 percent in math, 32 versus 31 percent in 
English/language arts).'® 


Even if support for DDI did not increase teachers’ use of specific data practices, professional 
development or coaching received in the intervention could have led teachers to use additional 
kinds of data or be more proficient in their data use. But we saw no evidence of this. Similar 
percentages of teachers in treatment and control schools reported having access to and using each 
of the specific types of student data we examined (appendix tables C.1 and C.2). We could not 
directly measure teachers’ data use proficiency, but we did measure how useful teachers found 
data and their confidence in their ability to use data to guide instruction. Similar percentages of 
teachers in the two groups reported finding different types of data “very useful” when making 
instructional decisions (appendix table C.3). And similar percentages of teachers in treatment (79 
percent) and control schools (77 percent) reported being confident or very confident in their data 
use abilities (appendix table C.4).!° 


a3 igures IV.1 and IV.2 measure data use that occurred at least several times per week because the expectation of 
DDI was that teachers should use data on a regular basis. We also examined whether teachers used data with a 
looser definition of what constitutes regular data use—at least weekly or several times per month. A larger 
proportion of teachers in both groups of schools—about 60 to 90 percent—reported using data for various activities 
this frequently (appendix figures C.1 and C.2). Teachers in treatment were more likely than those in control schools 
to use data for two of the nine activities in each subject, but similarly likely to use data for the other seven activities. 


Although data use by school principals is less directly connected with teachers’ instructional practices and student 
achievement than data use by teachers, we found that DDI increased selected types of data use among principals 
(appendix figure C.3). Those in treatment schools were more likely than those in control schools to use data to 
monitor student progress toward learning goals at least several times per month (92 versus 75 percent) as well as to 
determine the professional development needs of teachers (83 versus 60 percent). Principals also were more likely to 
report using certain types of data including interim assessments (appendix table C.5) and had fewer barriers to data 
use (appendix table C.6). These findings are consistent with findings reported in Chapter III that the intervention led 
to greater involvement of school leaders in supporting DDI. 
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Figure IV.1. Percentage of grade 4 and 5 teachers who reported using data 
daily or several times per week for different purposes in math during 2015- 
2016 
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Source: Teacher survey (n = 396-398). 
*Difference is statistically significant at the .05 level, two-tailed test. 


Figure IV.2. Percentage of grade 4 and 5 teachers who reported using data 
daily or several times per week for different purposes in English/language 
arts during 2015-2016 
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Source: Teacher survey (n = 408-411). 
*Difference is statistically significant at the .05 level, two-tailed test. 


Teachers’ Instructional Strategies 


Support for DDI did not increase teachers’ use of instructional strategies associated 
with DDI. Grade 4 and 5 teachers could potentially use data to alter their usual instructional 
approach by employing various strategies. We asked about five possible strategies they could 
have used more commonly as a result of DDI: (1) after-school tutoring for selected students, (2) 
additional instruction on specific topics, (3) new instructional approaches on difficult or complex 
concepts, (4) working with students to set and monitor individualized learning goals, and (5) 
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using pull-out or push-in services by a specialist. Similar percentages of teachers in treatment 
and control schools reported frequently—every day or several times per week—using each of 
these instructional strategies in both math and English/language arts (figure IV.3). In fact, the 
only significant difference indicates that teachers in treatment schools were /ess likely than those 
in control schools to frequently set individualized learning goals for students in English/language 
arts,” 


Figure IV.3. Percentage of grade 4 and 5 teachers who reported using 
instructional practice daily or several times per week for different purposes 
in English/language arts and math during 2015-2016 
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Source: Teacher survey (n = 397-411). 
ELA = English/language arts. 
*Difference is statistically significant at the .05 level, two-tailed test. 


Impacts of Support for DDI on Student Achievement 


Despite the fact that support for DDI did not affect teachers’ data use or instructional 
practices, the intervention could still have positively affected student achievement. For example, 
DDI may have improved the quality of teachers’ data use or instructional practices in ways not 
captured by the measures examined above, which focused on frequency of practices. In other 
words, the intervention could have led teachers to be more effective in using data to improve 
their existing instructional practices. In addition, schools had the freedom to tailor DDI for the 
particular needs of their students by selecting an area of instructional focus. DDI may have 
affected teacher practices and student achievement in a given subject, but only for schools that 
chose an area of instructional focus within that subject. 


More generally, even if DDI did not affect student achievement in the average school, it may 
have been successful for some students or in some contexts. In addition to examining how DDI 


0 This significant difference may not indicate an important impact of DDI on teachers’ instructional practices. 
When we examined teachers’ use of instructional practices using a looser definition of regular use of the practice— 
every week or several times per month, we found the reverse—that teachers in treatment schools were more likely 
than those in control schools to set individualized learning goals for students in English/language arts, with no other 
significant differences in either subject (appendix figure C.4). 
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affected student achievement in the average school, we also examined how the impacts of DDI 
differed for different groups of students or in different types of schools. In particular, we focused 
on whether the intervention was more successful in schools that seemed most well-prepared for 
DDI at the outset of the study, in terms of having the basic infrastructure in place. These schools 
might have had a useful head start in putting into place what was a complex intervention. 


Average Impacts on Student Achievement 


Support for DDI had no impact on student achievement in math or English/language 
arts. On average across the study sample, fourth- and fifth-grade students in treatment and 
control schools had similar achievement levels in math and English/language arts as measured by 
state end-of-year assessments in spring 2016 (figure IV.4).! In each subject, students in both 
groups of schools performed below the state average, which is consistent with the types of 
districts and schools the study team targeted during recruiting. In math, students in treatment 
schools were at the 39th percentile of the state distribution for their grade, whereas those in 
control schools were at the 41st percentile, on average. In English/language arts, both groups 
scored at the 40th percentile. These estimated impacts were equivalent to -0.04 student standard 
deviation units in math and -0.01 standard deviation units in English/language arts.” 


Figure IV.4. Mean student achievement on 2016 state assessments in math 
and English/language arts 
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Source: District student records (n = 12,018-12,036). 
Neither difference is statistically significant at the .05 level, two-tailed test. 


>! We also estimated the impact of DDI on student achievement in spring 2017 and found no evidence of impacts on 
achievement in that year. See Appendix Table C.7. 


22 The study was designed to reliably detect impacts of 0.12 standard deviations, based on an analysis of the study’s 
statistical power described in Appendix A (Table A.2). We also conducted a series of sensitivity tests and found that 
the estimated impacts of the DDI intervention were not sensitive to the approach used (appendix tables C.8 and C.9). 
These tests assessed whether attrition, our approach to controlling for students’ prior achievement, and the 
assumptions made in the impact model affected the results. We also examined student behavior, and found no 
evidence that DDI affected students’ school attendance or suspensions (appendix table C.10). 
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One concern about measuring the average impact of support for DDI on student 
achievement in a given subject across all schools is that different schools focused their DDI 
implementation activities on different subjects. As described in Chapter III, schools were free to 
choose an area of instructional focus, and most chose to focus on an area within 
English/language arts or within math, but not both. To allow for the possibility that DDI 
improved student achievement only in the area of instructional focus, we examined impacts on 
student achievement in a given subject only among those schools that had chosen an area of 
instructional focus within that subject alone. Because few schools chose to focus on math alone, 
we examined impacts on student achievement in English/language arts among schools that 
selected an area of instructional focus within English/language arts.** This analysis also 
examined impacts on student achievement in math among these schools, to determine whether 
the effects of the intervention may have spilled over into the subject that was not the area of 
focus. 


Support for DDI did not affect student achievement in English/language arts in the average 
school with an instructional focus in that subject. In these schools, students in both the treatment 
and control groups were at the 43rd percentile on the state distribution (figure IV.5). Although 
these scores were slightly higher than the score of students in the average study school overall, 
the impact of DDI on students’ English/language arts achievement was the same among schools 
that focused on this subject as it was among all study schools. Nor did DDI affect student 
achievement in math among schools with an instructional focus in English/language arts. 


3 This analysis did not include seven schools that chose areas of instructional focus in both math and reading, since 
the benefits of focusing implementation activities may have been diminished if multiple areas were selected in 
unrelated subjects. We also conducted a sensitivity analysis that examined the impacts of the intervention on student 
achievement among schools that focused either on English/language arts alone or on both English/language arts and 
math. The estimated impacts within that set of schools were similar to those within schools that focused on 
English/language arts alone (see appendix figure C.5). 
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Figure IV.5. Mean student achievement on 2016 state assessments in math 
and English/language arts, among schools with an instructional focus in an 
area of English/language arts 
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Source: District student records (n = 6,691 — 6,710). 
Neither difference is statistically significant at the .05 level, two-tailed test. 


We also examined whether support for DDI was more or less effective for particular kinds 
of students—whether there were differences in impacts among students in different grades or 
with different prior achievement levels. Impacts could have been stronger among fifth-graders 
than among fourth-graders in 2015-2016 because of the timing of DDI. The intervention began 
in December 2014 and targeted just those two grades. Thus, fifth-graders could have been 
affected by DDI in two school years (2014-2015 and 2015-2016), while fourth-graders could 
have been affected only in 2015-2016 since they would have been third graders in 2014-2015. 
We examined impacts separately for students with low versus high prior achievement because 
DDI activities focused largely on identifying struggling students and implementing instructional 
practices that could help those students. Thus, DDI may have been more successful for the low- 
achieving group.”* 


The impacts of support for DDI did not differ for students in different grades or with 
different levels of prior achievement. We found that the impacts of DDI on student 
achievement did not differ between fourth- and fifth-graders or between students at different 
levels of proficiency on their spring 2015 assessment (appendix figures C.6 and C.7). For each of 


a4 Alternatively, the intervention could have led teachers to target students near the threshold that distinguishes 
proficient and non-proficient students—sometimes called “bubble students.” Teachers aiming to increase the 
number of proficient students might have focused on this group if they believed that focusing on students well below 
proficiency would require too much effort. We did not investigate this hypothesis because prior achievement data 
were from the previous year’s state assessments, whereas teachers would likely have used interim assessments from 
the current year to identify bubble students. 
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these subgroups, the DDI intervention did not affect student achievement in math or 
English/language arts. 


Variation in the Impacts of Support for DDI across Schools 


While support for DDI did not affect student achievement in the average school, it may have 
been more successful in some schools than others. The intervention was complex, and successful 
implementation required both teachers and school leaders to be willing and able to put into place 
a number of different structures and practices related to DDI. As described in Chapter III, some 
study schools may have been unwilling or unable to put these structures in place, while others 
were slow to do so. In such schools, DDI would be unlikely to lead to improved student 
achievement by spring 2016. On the other hand, some schools already had the infrastructure 
required for DDI prior to the beginning of the intervention, and would not have to spend time 
doing so during the intervention period. This would leave the schools with more time for DDI 
activities—using data to better understand their students’ learning needs and identifying and 
implementing instructional strategies that might address those needs. Past research suggests that 
DDI is more likely to positively affect student achievement in schools with greater 
“implementation readiness” (West, Morton, and Herlihy 2016). 


We first examined whether there was meaningful variation in impacts across study schools. 
Because schools were matched into pairs and then randomly assigned, we could estimate the 
impacts of DDI separately in each matched pair of schools. Comparing how different these 
estimated impacts were in different study schools allowed us to assess variation in how DDI 
affected student achievement. The more the impacts differed from school to school, the greater 
the chances that the intervention may have been successful in an identifiable subset of schools, 
even though the average impacts were not positive. 


The impacts of support for DDI on students’ math and English/language arts 
achievement differed across schools. In both subjects, the estimated impacts of DDI ranged 
from about -1.0 to 0.6 standard deviations of student achievement (figures IV.6 and IV.7). As a 
reminder, the average impact across all schools was -0.04 in math and -0.01 in English/ language 
arts. After accounting for random variation across schools in estimated impacts, the actual 
variation between schools in the impacts of DDI on student achievement in each subject was 
statistically significant.*> There were also differences between districts in how DDI affected 
student achievement among schools in each subject (appendix figures C.8 and C.9). 


5 The range of impact estimates across schools was influenced both by true differences between schools in how 
DDI affected student achievement and by some random variation resulting from the fact that the impact in each 
matched pair was based on a comparison of student achievement in one treatment school versus one control school. 
The standard deviation of estimated matched-pair-level impacts was 0.29 in math and 0.30 in reading. Using a 
random-treatment effects model to distinguish the variation in matched-pair-level impacts from sampling error, we 
estimated that the true impacts of the DDI intervention on student test scores in each subject had a standard 
deviation of 0.21. 
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Figure IV.6. Estimated impact of Support for DDI on student achievement on 
2016 state assessments in math, by matched school pair 
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Source: District student records (n = 12,036). 
Variation in estimated impacts across matched pairs is statistically significant at the .05 level, two-tailed 
test. 
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Figure IV.7. Estimated impact of Support for DDI on student achievement on 
2016 state assessments in English/language arts, by matched school pair 


0.6 


. 
0.4 2 
$ ° 24 © 

2. 012 ° g 
w 
4 4 
‘< > e 4 
2 we g » © 2s & 
8 g © ° ° © e ° 
u 8 . ? * g ° ry 
# 02 © ° $ 
3 e ra . 
wn 
eg oo ° 
304 rm 8 
oO 
Qa 
E 

-0.6 

-0.8 

1 ° 
4 2 3 4 5 6 7 8 9 10 11 12 
District 
@ Impact in Treatment school © District-level impact estimate 


Source: District student records (n = 12,018). 
Variation in estimated impacts across matched pairs is statistically significant at the .05 level, two-tailed 
test. 


Given that another study of DDI (West, Morton, and Herlihy 2016) found impacts among 
schools with high levels of readiness, we next examined whether the positive impacts on student 
achievement were in schools that were more ready to implement DDI at the beginning of the 
intervention. Our measure of readiness was based on a count of four indicators of whether 
schools had key intervention structures in place prior to December 2014 (see box 4). A second 
measure of readiness required that schools have teacher collaboration teams regularly meeting to 
discuss data use and at least one of the other two remaining indicators in place previously.*° 


26 One caveat to this analysis is that we only could measure readiness in treatment schools and not in control 
schools. If readiness was related to student characteristics in ways that we did not account for in initially matching 
school pairs, the difference in impacts could have resulted from this preexisting difference between treatment and 
control schools in student characteristics. We assessed the likelihood of this possibility by measuring the treatment- 
control difference in schools’ 2013 test scores among those with high readiness to implement DDI versus those with 
low readiness. We found that treatment-control differences in 2013 scores were similar in matched pairs in the high- 
readiness group as in the matched pairs in the low-readiness group. This finding suggests that any differences in 
impact estimates between high- and low-readiness schools would not be caused by preexisting differences between 
students in treatment versus control schools. 
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Box 4. Were the impacts of support for DDI in a school related to 
implementation readiness? 


Because the DDI intervention included many different components and may have been challenging for schools to 
implement, we hypothesized that DDI had a larger impact in schools with greater implementation readiness at the 
beginning of the intervention. We defined readiness as having key school structures in place and having had 
some prior experience with data use in these structures. 


The implementation readiness measure was based on a count of four indicators. Each indicator measured 
whether a particular structure or practice was in place in the school before December 2014. These indicators 
were: 


(1) Presence of grade-level teacher collaboration teams in the school 

(2) Presence of grade-level teacher collaboration teams that had been regularly using data 
(3) Presence of an instructional leadership team 

(4) Presence of data walls (that is, visual displays of student data in the school) 


The primary measure of a school’s readiness to implement DDI defined schools to have high readiness if they 
had at least three of the four readiness indicators in place prior to December 2014, according to data coaches. 
Schools with two or fewer indicators were defined to have low readiness. Based on this measure, 27 percent of 
schools in the study sample were defined as high-readiness schools; the rest were defined as having low 
readiness. We estimated the impacts of DDI on student achievement separately for these two groups of schools. 


We also tested the sensitivity of the results to an alternative definition of DDI readiness based on the notion that 
the four indicators were not equally important. This definition of DDI readiness required that schools have teacher 
collaboration teams in place that engaged in data use to some extent prior to December 2014, based on the 
recommendation from Hamilton et al. (2009) that to implement DDI it is important for schools to “dedicate 
structured time for staff collaboration” and to have a “facilitator who meets with teacher teams to discuss data.” 
These schools also had to have at least one of the other two indicators in place to be defined as high-readiness 
schools. Schools with teacher collaboration teams that rarely or never used data were defined as low readiness 
schools, since any advantage to having teacher collaboration teams might be offset by the need to change the 
teams’ activities to focus primarily on data use (with the possibility that teachers might resist such a change). 
Based on this definition, 20 percent of schools in the study sample were defined as high-readiness schools. 
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Support for DDI did not consistently improve student achievement or increase 
teachers’ data use practices in schools with greater readiness to implement DDI. Based on 
the primary measure of DDI readiness, the estimated impact of DDI on students’ 
English/language arts achievement was 0.04 standard deviations in high-readiness schools and - 
0.05 in low-readiness schools, but not statistically significant in either group (figure IV.8).?’ In 
math, the estimated impact of DDI was negative in both groups of schools and statistically 
significant in low-readiness schools. Under the alternative readiness measure, there was a more 
striking difference in impacts between high- and low-readiness schools. For example, the 
estimated impact of DDI on students’ English/language arts achievement was 0.12 in high- 
readiness schools and -0.06 in low-readiness schools, and each estimate was statistically 
significant (appendix figure C.10). 


Since evidence that the intervention had positive impacts was strongest among schools 
defined as having high DDI readiness based on one of two definitions, we examined whether the 
intervention also affected the intermediate outcomes in these schools. We found that DDI did not 
lead to an increase in the measures of teachers’ practices, such as frequency of data use.”° In 
other words, these findings are not consistent with how the logic model would suggest that DDI 
leads to improved student outcomes. 


?7 The difference between high-readiness and low-readiness schools in the estimated impact of DDI on student 
achievement in English/language arts was marginally significant (p=0.10). 


°8 We measured the total number of data practices teachers reported frequently using out of the nine listed in Figures 
IV.1 and IV.2, as well as the total number of instructional practices associated with DDI out of the five listed in 
Figure IV.3. DDI was estimated to lead teachers to use 1.4 fewer data practices in math and 1.5 fewer data practices 
in English/language arts, although these impacts were not statistically significant. Similarly, the estimated impacts of 
the intervention on teachers use of instructional practices associated with DDI was negative but not significant. 
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Figure IV.8. Estimated impact of support for DDI on student achievement on 
2016 state assessments in math and English/ language arts, by school’s 
implementation readiness 
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Source: District student records (n = 9,693 — 9,708); data coach interviews. 
*Estimated impact within subgroup is statistically significant at the .05 level, two-tailed test. 
Neither difference in impact estimates between subgroups is statistically significant at the 0.05 level, two-tailed test. 


We also assessed whether support for DDI improved student achievement or teachers’ data 
use practices among schools in which coaches themselves appeared to have higher levels of 
readiness. We measured whether data coaches had any prior experience as a coach (although the 
vast majority had no experience as a data coach, 36 percent had prior experience as an 
instructional coach) and whether they reported that the training they received through DDI 
prepared them to carry out most or all of the tasks associated with being a data coach. Neither of 
these measures consistently identified schools with more positive impacts on student 
achievement and teacher practices (appendix tables C.11 and C.12). In some cases, prior 
experience as an instructional coach was associated with negative impacts, while having no 
experience was associated with positive impacts. (The number of teachers with prior experience 
as a data coach was too small to analyze the association between impacts on outcomes and the 
most relevant type of prior experience.) Having a higher level of self-reported preparation was 
not significantly associated with any outcomes, but having a lower level was linked to some 
negative outcomes. Since neither of these dimensions of coach readiness were randomly 
assigned, their relationships with impacts could be due to other factors related to coach readiness, 
and so these findings should be interpreted with caution. 
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Concluding Thoughts 


The professional development program to support the intervention examined in this report 
was built on the idea that the key to the success of DDI was to provide teachers with various 
forms of support to help them use data to improve their instruction. This support would come 
from data coaches, school leaders, and other teachers. According to the study’s logic model, this 
support would enable and encourage teachers to more frequently use data to better understand 
their students’ learning needs and identify instructional practices to better address those needs. 
For example, they might more frequently use differentiated instruction or spend more time on 
concepts their students were struggling to understand. These practices would, in turn, lead to 
improved student achievement. 


Results from the study’s implementation analysis showed that the intervention succeeded in 
providing more support to teachers in the form of data-related professional development, data 
coaching, and guidance from school leaders. While teachers in control schools did receive some 
of these supports for data use, those in treatment schools received significantly more supports. 
However, the study’s impact analysis showed that this additional support did not lead to more 
frequent data use by teachers, nor did it lead to more frequent use of the instructional practices 
associated with DDI. Consistent with those findings, the support for DDI intervention did not 
lead to improved student achievement. 


These results are consistent with other recent studies focused on giving teachers support in 
analyzing student data to improve instruction and student achievement. The results suggest either 
that the intervention described in the study’s logic model was not effectively implemented, or 
that the theory underlying the logic model was wrong or incomplete. With respect to 
implementation, the study found the professional development sessions were held as planned but 
did not assess the quality of these sessions. Nor did the study assess the quality of data coaches 
or their content knowledge, although we did find that they had little previous experience as data 
coaches. Within schools, implementation was effective in engaging school leaders, but less 
effective in promoting teacher collaboration around analyzing or using data. The teams 
collaborated on data no more frequently than in control schools. One challenge to the 
effectiveness of the teacher collaboration teams may have been teachers’ other responsibilities in 
the school. Data coaches reported difficulties within these teams related to follow-through on 
meeting decisions and balancing DDI work with other responsibilities, challenges consistent with 
other DDI interventions and likely to be present in most schools that might attempt to implement 
DDI (Cavalluzzo et al. 2014; Slavin et al. 2013).?? On balance, however, the study did not 
identify major problems with the implementation of the support for DDI intervention in study 
schools. 


Given this, one possible explanation for the absence of impacts on teachers’ data use or 
associated instructional practices is that even with successful implementation of study supports, 
teachers experienced difficulty engaging in these practices during the study period. From the 
perspective of teachers, the intervention was complex and had a lot of moving pieces. Teachers 
had to analyze student data, diagnose potential issues in their instructional practices based on this 
data analysis, identify more effective approaches, and put them into place in their classrooms. 


2° Some of the additional challenges noted in the literature include staff turnover (Cavalluzo et al. 2014), a lack of 
capacity to implement training sessions and assessments (Cordray et al. 2012), and teachers’ lack of flexibility to 
reorganize their instruction in order to account for areas of need in their student data (West et al. 2016). 
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The difficulty individuals experience when adopting new approaches to their work, sometimes 
referred to as the “implementation dip” (Fullan 2007) or “performance dip” (Eastwood and Louis 
1992), could explain the lack of impacts on key teacher practices encouraged by DDI supports. 
The approach examined in this study of providing support for DDI may have required more time 
than the year-and-a-half follow-up period teachers had to put it into place in their classrooms. 


The theory underlying the logic model emphasized the importance of teachers being 
supported in their data use, and the intervention accordingly focused on facilitating that 
support.*” While teachers in treatment schools did receive more supports than those in control 
schools, another possible explanation for the lack of impacts is that this contrast between the 
supports received by treatment and control schools was not large enough to affect teachers’ 
instructional practices or student achievement. For example, teachers in one-third of control 
schools also had data coaches. While teachers in nearly all treatment schools had data coaches, 
few of them had previous experience in that role. Thus, the coaching received by teachers in the 
two groups of schools may not have been different enough to lead to better outcomes in the 
treatment group. 


The logic model also implied that the work of selecting instructional practices appropriate 
for individual teachers would occur largely within teacher collaboration teams. The intervention 
focused less on identifying the specific instructional practices teachers should put into place 
given their circumstances. Thus, one possible limitation of the theory behind the logic model is 
that while providing general support and guidance to teachers is important, it may not be 
sufficient. Teachers may also need more specific expert input into which instructional strategies 
would be most effective for them in light of their student data. If so, then future work in 
developing DDI interventions could explore different ways of providing teachers with this input 
on exactly how they should change their instructional practices. This study shows that an 
intervention that focused primarily on providing additional support to teachers from trained data 
coaches and school leaders was not enough to improve student achievement. 


3° Since all study districts received student data, the study was not designed to test whether providing a new or 
different form of student data would be the key to the success of DDI. 
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